[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:15:38 dist-test-slave-vnbh krb5kdc[32230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:15:38 dist-test-slave-vnbh krb5kdc[32230](info): set up 2 sockets
Sep 26 20:15:38 dist-test-slave-vnbh krb5kdc[32230](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 20:15:39.054404 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:15:39 dist-test-slave-vnbh krb5kdc[32230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:15:39.189935 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:44759
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:44759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:39.307958 32243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:39.308323 32243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:39.308413 32243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:39.312804 32243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:39.312915 32243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:39.312950 32243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:39.312990 32243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:39.313030 32243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:39.319105 32243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:44759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:44759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:39.320359 32243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:39.321578 32243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:39.331880 32248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:39.331969 32243 server_base.cc:1034] running on GCE node
W20260926 20:15:39.331948 32249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:39.332048 32251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:39.333070 32243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:39.334546 32243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:39.335879 32243 hybrid_clock.cc:648] HybridClock initialized: now 1790453739335867 us; error 37 us; skew 500 ppm
Sep 26 20:15:39 dist-test-slave-vnbh krb5kdc[32230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:39.339330 32243 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:15:39.340516 32243 webserver.cc:460] Webserver started at http://127.31.119.254:40667/ using document root <none> and password file <none>
I20260926 20:15:39.341157 32243 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:39.341261 32243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:39.341563 32243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:39.343782 32243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "85b3735c6b734b389c38fd6d804255b9"
format_stamp: "Formatted at 2026-09-26 20:15:39 on dist-test-slave-vnbh"
server_key: "258ef16fe94c5ee957c0dd804352e2ed"
server_key_iv: "ce23d33a3ac21d62ac0ea554472b5d70"
server_key_version: "encryptionkey@0"
I20260926 20:15:39.344439 32243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "85b3735c6b734b389c38fd6d804255b9"
format_stamp: "Formatted at 2026-09-26 20:15:39 on dist-test-slave-vnbh"
server_key: "258ef16fe94c5ee957c0dd804352e2ed"
server_key_iv: "ce23d33a3ac21d62ac0ea554472b5d70"
server_key_version: "encryptionkey@0"
I20260926 20:15:39.348860 32243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 20:15:39.351719 32258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:39.353398 32243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:15:39.353565 32243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "85b3735c6b734b389c38fd6d804255b9"
format_stamp: "Formatted at 2026-09-26 20:15:39 on dist-test-slave-vnbh"
server_key: "258ef16fe94c5ee957c0dd804352e2ed"
server_key_iv: "ce23d33a3ac21d62ac0ea554472b5d70"
server_key_version: "encryptionkey@0"
I20260926 20:15:39.353794 32243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:39.395896 32243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:39.399955 32243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:39.400195 32243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:39.407696 32243 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:44759
I20260926 20:15:39.407733 32310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:44759 every 8 connection(s)
I20260926 20:15:39.408829 32243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:15:39.411916 32311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:39.417702 32311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9: Bootstrap starting.
I20260926 20:15:39.418332 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 32243
I20260926 20:15:39.418520 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:15:39.419893 32223 external_mini_cluster.cc:1428] Setting key 0fa4db45c36674c37deaf7aa6978c8c7
I20260926 20:15:39.420580 32311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:39.421413 32311 log.cc:824] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:39.424110 32311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9: No bootstrap required, opened a new log
I20260926 20:15:39.428333 32311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b3735c6b734b389c38fd6d804255b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44759 } }
I20260926 20:15:39.428601 32311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:39.428660 32311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b3735c6b734b389c38fd6d804255b9, State: Initialized, Role: FOLLOWER
I20260926 20:15:39.429239 32311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b3735c6b734b389c38fd6d804255b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44759 } }
I20260926 20:15:39.429450 32311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:15:39.429543 32311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:15:39.429664 32311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:39.430641 32311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b3735c6b734b389c38fd6d804255b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44759 } }
I20260926 20:15:39.430999 32311 leader_election.cc:302] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85b3735c6b734b389c38fd6d804255b9; no voters:
I20260926 20:15:39.431349 32311 leader_election.cc:288] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:15:39.431419 32316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:39.431660 32316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [term 1 LEADER]: Becoming Leader. State: Replica: 85b3735c6b734b389c38fd6d804255b9, State: Running, Role: LEADER
I20260926 20:15:39.431964 32316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b3735c6b734b389c38fd6d804255b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44759 } }
I20260926 20:15:39.432704 32311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:39.433631 32318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b3735c6b734b389c38fd6d804255b9. Latest consensus state: current_term: 1 leader_uuid: "85b3735c6b734b389c38fd6d804255b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b3735c6b734b389c38fd6d804255b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44759 } } }
I20260926 20:15:39.433826 32318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:39.435210 32317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85b3735c6b734b389c38fd6d804255b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b3735c6b734b389c38fd6d804255b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44759 } } }
I20260926 20:15:39.435595 32317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9 [sys.catalog]: This master's current role is: LEADER
W20260926 20:15:39.438563 32331 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:15:39.438642 32331 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:15:39.438719 32325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:15:39.439634 32325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 20:15:39 dist-test-slave-vnbh krb5kdc[32230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:39.446081 32325 catalog_manager.cc:1347] Generated new cluster ID: 9ebe7086cfc5450bba031a76aa623b53
I20260926 20:15:39.446156 32325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:15:39.447693 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:39.423334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38042 (local address 127.31.119.254:44759)
0926 20:15:39.423834 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:39.423851 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:39.425312 (+ 1461us) server_negotiation.cc:407] Connection header received
0926 20:15:39.428055 (+ 2743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:39.428074 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:39.428387 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:39.428735 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:39.431327 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:39.432645 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:39.434462 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:39.435243 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:39.435412 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:39.441078 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:39.441084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:39.441102 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:39.441137 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:39.443396 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:39.443986 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:39.443992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:39.443998 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:39.444179 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:39.444691 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:39.444694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:39.444696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:39.445320 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:39.445552 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:39.446346 (+ 794us) server_negotiation.cc:299] Negotiation successful
0926 20:15:39.446696 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":145,"threads_started":1}
I20260926 20:15:39.458274 32325 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:15:39.459321 32325 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:15:39.469762 32325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85b3735c6b734b389c38fd6d804255b9: Generated new TSK 0
I20260926 20:15:39.470624 32325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:15:39.505123 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44759
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:39.626905 32337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:39.627244 32337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:39.627295 32337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:39.632448 32337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:39.632544 32337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:39.632673 32337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:15:39.640502 32337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44759
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:39.642007 32337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:39.643219 32337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:39.654167 32345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:39.654167 32342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:39.654211 32343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:39.654361 32337 server_base.cc:1034] running on GCE node
I20260926 20:15:39.655105 32337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:39.655835 32337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:39.657017 32337 hybrid_clock.cc:648] HybridClock initialized: now 1790453739657002 us; error 28 us; skew 500 ppm
Sep 26 20:15:39 dist-test-slave-vnbh krb5kdc[32230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:39.660332 32337 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:15:39.661409 32337 webserver.cc:460] Webserver started at http://127.31.119.193:38461/ using document root <none> and password file <none>
I20260926 20:15:39.662091 32337 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:39.662194 32337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:39.662448 32337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:39.664646 32337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "8836c632783b47ffb899b57b0226afb9"
format_stamp: "Formatted at 2026-09-26 20:15:39 on dist-test-slave-vnbh"
server_key: "144206c52dfb19f6836d280ac95872f5"
server_key_iv: "9472636961e2185239026184f32cae25"
server_key_version: "encryptionkey@0"
I20260926 20:15:39.665249 32337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "8836c632783b47ffb899b57b0226afb9"
format_stamp: "Formatted at 2026-09-26 20:15:39 on dist-test-slave-vnbh"
server_key: "144206c52dfb19f6836d280ac95872f5"
server_key_iv: "9472636961e2185239026184f32cae25"
server_key_version: "encryptionkey@0"
I20260926 20:15:39.669445 32337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:15:39.672307 32352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:39.673930 32337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:15:39.674093 32337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "8836c632783b47ffb899b57b0226afb9"
format_stamp: "Formatted at 2026-09-26 20:15:39 on dist-test-slave-vnbh"
server_key: "144206c52dfb19f6836d280ac95872f5"
server_key_iv: "9472636961e2185239026184f32cae25"
server_key_version: "encryptionkey@0"
I20260926 20:15:39.674239 32337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:39.689415 32337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:39.693387 32337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:39.693610 32337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:39.694280 32337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:15:39.695189 32337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:39.695264 32337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:39.695351 32337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:39.695395 32337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:39.705137 32337 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:43617
I20260926 20:15:39.705176 32465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:43617 every 8 connection(s)
I20260926 20:15:39.706187 32337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:15:39.712548 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 32337
I20260926 20:15:39.712689 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:15:39.712953 32223 external_mini_cluster.cc:1428] Setting key 3e682cef07d133dca9470220e37258df
Sep 26 20:15:39 dist-test-slave-vnbh krb5kdc[32230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:39.723671 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:39.708404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:57761 (local address 127.31.119.254:44759)
0926 20:15:39.708570 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:39.708575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:39.709587 (+ 1012us) server_negotiation.cc:407] Connection header received
0926 20:15:39.711084 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:39.711089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:39.711153 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:39.711252 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:39.712796 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:39.713516 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:39.715065 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:39.715292 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:39.715384 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:39.718497 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:39.718504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:39.718511 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:39.718551 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:39.720758 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:39.721501 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:39.721506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:39.721508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:39.721622 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:39.722079 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:39.722084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:39.722098 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:39.722405 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:39.722539 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:39.723238 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 20:15:39.723406 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
I20260926 20:15:39.724536 32468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:39.708884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:44759 (local address 127.31.119.193:57761)
0926 20:15:39.709431 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:39.709478 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:39.710808 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:39.711420 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:39.711427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:39.711872 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:39.712498 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:39.712522 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:39.713726 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:39.713730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:39.714928 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:39.714942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:39.715462 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:39.715469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:39.715725 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:39.716506 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:39.716524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:39.718299 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:39.720936 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:39.720942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:39.720958 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:39.721378 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:39.721770 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:39.721773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:39.721774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:39.721944 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:39.722563 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:39.722570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:39.722915 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 20:15:39.723092 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 20:15:39.723390 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260926 20:15:39.726446 32466 heartbeater.cc:347] Connected to a master server at 127.31.119.254:44759
I20260926 20:15:39.726804 32466 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
I20260926 20:15:39.727517 32466 heartbeater.cc:511] Master 127.31.119.254:44759 requested a full tablet report, sending...
I20260926 20:15:39.729476 32275 ts_manager.cc:194] Registered new tserver with Master: 8836c632783b47ffb899b57b0226afb9 (127.31.119.193:43617)
I20260926 20:15:39.730971 32275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:57761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:15:39.744572 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44759
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:39.859063 32471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:39.859403 32471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:39.859499 32471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:39.863713 32471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:39.863819 32471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:39.863914 32471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:15:39.869593 32471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44759
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:39.870783 32471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:39.871869 32471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:39.882143 32477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:39.882177 32479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:39.882339 32471 server_base.cc:1034] running on GCE node
W20260926 20:15:39.882134 32476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:39.882876 32471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:39.883502 32471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:39.884721 32471 hybrid_clock.cc:648] HybridClock initialized: now 1790453739884655 us; error 76 us; skew 500 ppm
Sep 26 20:15:39 dist-test-slave-vnbh krb5kdc[32230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:39.887669 32471 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:15:39.888571 32471 webserver.cc:460] Webserver started at http://127.31.119.194:34595/ using document root <none> and password file <none>
I20260926 20:15:39.889201 32471 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:39.889266 32471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:39.889554 32471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:39.891775 32471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "44dfeb724d8e459088ecefe85a9ed0d7"
format_stamp: "Formatted at 2026-09-26 20:15:39 on dist-test-slave-vnbh"
server_key: "1e19b0c9c28e806bdf96958ee9fb547f"
server_key_iv: "ea769cc61acea319d807354d85d08fef"
server_key_version: "encryptionkey@0"
I20260926 20:15:39.892365 32471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "44dfeb724d8e459088ecefe85a9ed0d7"
format_stamp: "Formatted at 2026-09-26 20:15:39 on dist-test-slave-vnbh"
server_key: "1e19b0c9c28e806bdf96958ee9fb547f"
server_key_iv: "ea769cc61acea319d807354d85d08fef"
server_key_version: "encryptionkey@0"
I20260926 20:15:39.896817 32471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 20:15:39.899698 32486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:39.901173 32471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:39.901352 32471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "44dfeb724d8e459088ecefe85a9ed0d7"
format_stamp: "Formatted at 2026-09-26 20:15:39 on dist-test-slave-vnbh"
server_key: "1e19b0c9c28e806bdf96958ee9fb547f"
server_key_iv: "ea769cc61acea319d807354d85d08fef"
server_key_version: "encryptionkey@0"
I20260926 20:15:39.901489 32471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:39.913810 32471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:39.917928 32471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:39.918145 32471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:39.918768 32471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:15:39.919654 32471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:39.919703 32471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:39.919744 32471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:39.919759 32471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:39.929127 32471 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:44385
I20260926 20:15:39.929167 32599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:44385 every 8 connection(s)
I20260926 20:15:39.930158 32471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:15:39.931677 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 32471
I20260926 20:15:39.931772 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:15:39.932106 32223 external_mini_cluster.cc:1428] Setting key 34339ae3e8a4aa41f5bcbfa4c3d17e55
Sep 26 20:15:39 dist-test-slave-vnbh krb5kdc[32230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453739, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
I20260926 20:15:39.946276 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:39.932199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:38827 (local address 127.31.119.254:44759)
0926 20:15:39.932372 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:39.932377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:39.933471 (+ 1094us) server_negotiation.cc:407] Connection header received
0926 20:15:39.935122 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:39.935126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:39.935194 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:39.935268 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:39.937161 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:39.937798 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:39.938995 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:39.939238 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:39.939364 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:39.942196 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:39.942199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:39.942201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:39.942229 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:39.943649 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:39.944277 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:39.944283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:39.944285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:39.944371 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:39.944794 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:39.944797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:39.944798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:39.945053 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:39.945179 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:39.945866 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 20:15:39.946021 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:15:39.947259 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:39.932635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:44759 (local address 127.31.119.194:38827)
0926 20:15:39.933314 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:39.933377 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:39.934870 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:39.935582 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:39.935590 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:39.936221 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:39.937006 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:39.937024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:39.937932 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:39.937935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:39.938861 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:39.938870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:39.939420 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:39.939425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:39.939634 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:39.940310 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:39.940327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:39.941993 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:39.943849 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:39.943855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:39.943873 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:39.944159 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:39.944478 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:39.944481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:39.944483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:39.944687 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:39.945180 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:39.945186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:39.945525 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 20:15:39.945781 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 20:15:39.946081 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":160,"threads_started":1}
I20260926 20:15:39.948637 32600 heartbeater.cc:347] Connected to a master server at 127.31.119.254:44759
I20260926 20:15:39.948922 32600 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:39.949631 32600 heartbeater.cc:511] Master 127.31.119.254:44759 requested a full tablet report, sending...
I20260926 20:15:39.950875 32275 ts_manager.cc:194] Registered new tserver with Master: 44dfeb724d8e459088ecefe85a9ed0d7 (127.31.119.194:44385)
I20260926 20:15:39.951488 32275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:38827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:15:39.964339 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44759
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:40.081429 32605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:40.081727 32605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:40.081780 32605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:40.086082 32605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:40.086163 32605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:40.086228 32605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:15:40.092186 32605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44759
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:40.093439 32605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:40.094776 32605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:15:40.105675 32605 server_base.cc:1034] running on GCE node
W20260926 20:15:40.105669 32613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:40.105643 32610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:40.105671 32611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:40.106475 32605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:40.107127 32605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:40.108325 32605 hybrid_clock.cc:648] HybridClock initialized: now 1790453740108279 us; error 58 us; skew 500 ppm
Sep 26 20:15:40 dist-test-slave-vnbh krb5kdc[32230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453740, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:40.111512 32605 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:15:40.112537 32605 webserver.cc:460] Webserver started at http://127.31.119.195:46481/ using document root <none> and password file <none>
I20260926 20:15:40.113164 32605 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:40.113240 32605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:40.113497 32605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:40.115715 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "356a2a71397e40eb9ac37ac44d54192f"
format_stamp: "Formatted at 2026-09-26 20:15:40 on dist-test-slave-vnbh"
server_key: "86b1c285142e8d1fbcb629b4c655ab93"
server_key_iv: "968c54e5962a26f306e0462e94b5cfd6"
server_key_version: "encryptionkey@0"
I20260926 20:15:40.116377 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "356a2a71397e40eb9ac37ac44d54192f"
format_stamp: "Formatted at 2026-09-26 20:15:40 on dist-test-slave-vnbh"
server_key: "86b1c285142e8d1fbcb629b4c655ab93"
server_key_iv: "968c54e5962a26f306e0462e94b5cfd6"
server_key_version: "encryptionkey@0"
I20260926 20:15:40.120641 32605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:15:40.123690 32620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:40.125912 32605 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:15:40.126093 32605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "356a2a71397e40eb9ac37ac44d54192f"
format_stamp: "Formatted at 2026-09-26 20:15:40 on dist-test-slave-vnbh"
server_key: "86b1c285142e8d1fbcb629b4c655ab93"
server_key_iv: "968c54e5962a26f306e0462e94b5cfd6"
server_key_version: "encryptionkey@0"
I20260926 20:15:40.126245 32605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:40.160806 32605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:40.164697 32605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:40.164943 32605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:40.165668 32605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:15:40.166635 32605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:40.166685 32605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:40.166738 32605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:40.166795 32605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:40.176395 32605 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:35701
I20260926 20:15:40.176409 32733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:35701 every 8 connection(s)
I20260926 20:15:40.177488 32605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:15:40.181739 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 32605
I20260926 20:15:40.181887 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:15:40.182228 32223 external_mini_cluster.cc:1428] Setting key ac9be8af3e04a735969c039eec7f81b9
Sep 26 20:15:40 dist-test-slave-vnbh krb5kdc[32230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453740, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:40.193585 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:40.179503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:54353 (local address 127.31.119.254:44759)
0926 20:15:40.179697 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:40.179710 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:40.180531 (+ 821us) server_negotiation.cc:407] Connection header received
0926 20:15:40.182118 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:40.182123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:40.182186 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:40.182292 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:40.184288 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:40.184956 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:40.186309 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:40.186543 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:40.186638 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:40.189649 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:40.189652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:40.189654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:40.189682 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:40.191324 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:40.191933 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:40.191936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:40.191937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:40.192019 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:40.192435 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:40.192438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:40.192439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:40.192639 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:40.192718 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:40.193269 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 20:15:40.193458 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 20:15:40.194523 32736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:40.179921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:44759 (local address 127.31.119.195:54353)
0926 20:15:40.180375 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:40.180420 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:40.181824 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:40.182663 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:40.182673 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:40.183320 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:40.184125 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:40.184158 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:40.185125 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:40.185129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:40.186155 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:40.186166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:40.186734 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:40.186738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:40.186911 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:40.187693 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:40.187716 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:40.189457 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:40.191481 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:40.191487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:40.191503 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:40.191817 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:40.192130 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:40.192133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:40.192135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:40.192319 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:40.192745 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:40.192753 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:40.193034 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 20:15:40.193204 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 20:15:40.193618 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260926 20:15:40.196156 32734 heartbeater.cc:347] Connected to a master server at 127.31.119.254:44759
I20260926 20:15:40.196480 32734 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:40.197146 32734 heartbeater.cc:511] Master 127.31.119.254:44759 requested a full tablet report, sending...
I20260926 20:15:40.198348 32275 ts_manager.cc:194] Registered new tserver with Master: 356a2a71397e40eb9ac37ac44d54192f (127.31.119.195:35701)
I20260926 20:15:40.198949 32275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:54353
I20260926 20:15:40.209964 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:15:40 dist-test-slave-vnbh krb5kdc[32230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:15:40 dist-test-slave-vnbh krb5kdc[32230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453740, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:15:40.238865 32741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:40.227635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.31.119.193:43617)
0926 20:15:40.228035 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:40.228042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:40.228068 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:15:40.228121 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:40.228126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:40.228356 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:40.228502 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:40.229348 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:40.230163 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:40.231177 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:40.231738 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:40.231838 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:40.235207 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:40.235219 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:40.235235 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:40.235265 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:40.237004 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:40.237508 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:40.237514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:40.237516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:40.237604 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:40.237931 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:40.237933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:40.237934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:40.238214 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:40.238373 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:40.238540 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 20:15:40.238726 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":177,"threads_started":1}
W20260926 20:15:40.240008 32380 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56382
Sep 26 20:15:40 dist-test-slave-vnbh krb5kdc[32230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:15:40 dist-test-slave-vnbh krb5kdc[32230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:15:40.268369 32741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:40.259153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.31.119.193:43617)
0926 20:15:40.259386 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:40.259391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:40.259535 (+ 144us) server_negotiation.cc:407] Connection header received
0926 20:15:40.259751 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:40.259755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:40.259802 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:40.259900 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:40.260636 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:40.261147 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:40.262030 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:40.262262 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:40.262372 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:40.265093 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:40.265096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:40.265098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:40.265125 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:40.266614 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:40.267103 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:40.267106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:40.267108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:40.267187 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:40.267446 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:40.267448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:40.267449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:40.267616 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:40.267739 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:40.267975 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 20:15:40.268080 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260926 20:15:40.269560 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 32337
I20260926 20:15:40.275578 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 32471
I20260926 20:15:40.283191 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 32605
I20260926 20:15:40.288973 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 32243
2026-09-26T20:15:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2310 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:15:40 dist-test-slave-vnbh krb5kdc[32748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:15:40 dist-test-slave-vnbh krb5kdc[32748](info): set up 2 sockets
Sep 26 20:15:40 dist-test-slave-vnbh krb5kdc[32748](info): commencing operation
krb5kdc: starting...
W20260926 20:15:41.349905 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/test-user.keytab.
W20260926 20:15:42.053983 32223 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.639s user 0.000s sys 0.002s
Sep 26 20:15:42 dist-test-slave-vnbh krb5kdc[32748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:15:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:15:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
W20260926 20:15:43.067236 32223 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.119.254: real 0.989s user 0.002s sys 0.001s
I20260926 20:15:43.068091 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:33799
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:33799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:43.194156 32761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:43.194494 32761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:43.194540 32761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:43.199002 32761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:43.199090 32761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:43.199110 32761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:43.199131 32761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:43.199146 32761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:43.205271 32761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:33799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:43.206643 32761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:43.207736 32761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:43.217567 32766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:43.217545 32767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:43.217689 32761 server_base.cc:1034] running on GCE node
W20260926 20:15:43.217983 301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:43.218712 32761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:43.220062 32761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:43.221323 32761 hybrid_clock.cc:648] HybridClock initialized: now 1790453743221263 us; error 36 us; skew 500 ppm
Sep 26 20:15:43 dist-test-slave-vnbh krb5kdc[32748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453743, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:43.229575 32761 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:15:43.230811 32761 webserver.cc:460] Webserver started at http://127.31.119.254:36399/ using document root <none> and password file <none>
I20260926 20:15:43.231490 32761 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:43.231603 32761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:43.231838 32761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:43.234638 32761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "e4e9263ddcac443c9abd77ab02bcbc38"
format_stamp: "Formatted at 2026-09-26 20:15:43 on dist-test-slave-vnbh"
server_key: "a81581787be9216835bee918bffca03d"
server_key_iv: "f31303cce59b9e796dc1671582518943"
server_key_version: "encryptionkey@0"
I20260926 20:15:43.235318 32761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "e4e9263ddcac443c9abd77ab02bcbc38"
format_stamp: "Formatted at 2026-09-26 20:15:43 on dist-test-slave-vnbh"
server_key: "a81581787be9216835bee918bffca03d"
server_key_iv: "f31303cce59b9e796dc1671582518943"
server_key_version: "encryptionkey@0"
I20260926 20:15:43.239959 32761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:15:43.242938 308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:43.244429 32761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:15:43.244623 32761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "e4e9263ddcac443c9abd77ab02bcbc38"
format_stamp: "Formatted at 2026-09-26 20:15:43 on dist-test-slave-vnbh"
server_key: "a81581787be9216835bee918bffca03d"
server_key_iv: "f31303cce59b9e796dc1671582518943"
server_key_version: "encryptionkey@0"
I20260926 20:15:43.244854 32761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:43.262017 32761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:43.266855 32761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:43.267104 32761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:43.276185 32761 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:33799
I20260926 20:15:43.276306 360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:33799 every 8 connection(s)
I20260926 20:15:43.277479 32761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:15:43.280637 361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:43.286366 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 32761
I20260926 20:15:43.286489 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:15:43.286769 32223 external_mini_cluster.cc:1428] Setting key 823fab5251c30b421f94c33295d68a17
I20260926 20:15:43.286937 361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38: Bootstrap starting.
I20260926 20:15:43.289801 361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:43.290737 361 log.cc:824] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:43.293118 361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38: No bootstrap required, opened a new log
Sep 26 20:15:43 dist-test-slave-vnbh krb5kdc[32748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:43.297641 361 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e9263ddcac443c9abd77ab02bcbc38" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33799 } }
I20260926 20:15:43.297976 361 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:43.298034 361 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e9263ddcac443c9abd77ab02bcbc38, State: Initialized, Role: FOLLOWER
I20260926 20:15:43.298712 361 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e9263ddcac443c9abd77ab02bcbc38" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33799 } }
I20260926 20:15:43.298898 361 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:15:43.299000 361 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:15:43.299111 361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:43.302943 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:43.288385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.31.119.254:33799)
0926 20:15:43.288896 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:43.288916 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:43.288956 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:15:43.289776 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:43.289799 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:43.290296 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:43.290651 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:43.291650 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:43.292677 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:43.293681 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:43.294192 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:43.294389 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:43.297523 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:43.297529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:43.297547 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:43.297580 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:43.299777 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:43.300288 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:43.300294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:43.300301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:43.300431 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:43.300735 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:43.300739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:43.300741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:43.301176 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:43.301433 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:43.301746 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 20:15:43.302110 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":181,"threads_started":1}
I20260926 20:15:43.366036 361 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e9263ddcac443c9abd77ab02bcbc38" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33799 } }
I20260926 20:15:43.366670 361 leader_election.cc:302] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4e9263ddcac443c9abd77ab02bcbc38; no voters:
I20260926 20:15:43.367161 361 leader_election.cc:288] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:15:43.367344 366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:43.367630 366 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [term 1 LEADER]: Becoming Leader. State: Replica: e4e9263ddcac443c9abd77ab02bcbc38, State: Running, Role: LEADER
I20260926 20:15:43.368070 366 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e9263ddcac443c9abd77ab02bcbc38" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33799 } }
I20260926 20:15:43.368916 361 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:43.369632 367 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4e9263ddcac443c9abd77ab02bcbc38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e9263ddcac443c9abd77ab02bcbc38" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33799 } } }
I20260926 20:15:43.369773 367 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:43.370201 370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:15:43.370489 368 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4e9263ddcac443c9abd77ab02bcbc38. Latest consensus state: current_term: 1 leader_uuid: "e4e9263ddcac443c9abd77ab02bcbc38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e9263ddcac443c9abd77ab02bcbc38" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33799 } } }
I20260926 20:15:43.370643 368 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:43.373838 370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:15:43.380133 370 catalog_manager.cc:1347] Generated new cluster ID: 5d47d7d3da5444efb743729597ffdb9f
I20260926 20:15:43.380227 370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:15:43.389252 370 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:15:43.390360 370 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:15:43.398170 370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4e9263ddcac443c9abd77ab02bcbc38: Generated new TSK 0
I20260926 20:15:43.398979 370 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
W20260926 20:15:44.619387 32223 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.119.193: real 1.207s user 0.000s sys 0.004s
I20260926 20:15:44.619902 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:33799
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:44.739828 387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:44.740147 387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:44.740195 387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:44.744470 387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:44.744555 387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:44.744626 387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:15:44.750869 387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:33799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:44.752002 387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:44.753075 387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:44.764245 392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:44.764307 387 server_base.cc:1034] running on GCE node
W20260926 20:15:44.764261 393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:44.764262 395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:44.765018 387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:44.765789 387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:44.766981 387 hybrid_clock.cc:648] HybridClock initialized: now 1790453744766947 us; error 46 us; skew 500 ppm
Sep 26 20:15:44 dist-test-slave-vnbh krb5kdc[32748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453744, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:44.770529 387 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:15:44.771765 387 webserver.cc:460] Webserver started at http://127.31.119.193:37339/ using document root <none> and password file <none>
I20260926 20:15:44.772392 387 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:44.772490 387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:44.772768 387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:44.775059 387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "645951120a0245c89b6f957378648e04"
format_stamp: "Formatted at 2026-09-26 20:15:44 on dist-test-slave-vnbh"
server_key: "c23e5cf0b79e462b7a6ff17669dfddf9"
server_key_iv: "8579da4ebe3dddfba6a5f0b309786a85"
server_key_version: "encryptionkey@0"
I20260926 20:15:44.775854 387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "645951120a0245c89b6f957378648e04"
format_stamp: "Formatted at 2026-09-26 20:15:44 on dist-test-slave-vnbh"
server_key: "c23e5cf0b79e462b7a6ff17669dfddf9"
server_key_iv: "8579da4ebe3dddfba6a5f0b309786a85"
server_key_version: "encryptionkey@0"
I20260926 20:15:44.780200 387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:15:44.783288 402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:44.784756 387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:44.784925 387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "645951120a0245c89b6f957378648e04"
format_stamp: "Formatted at 2026-09-26 20:15:44 on dist-test-slave-vnbh"
server_key: "c23e5cf0b79e462b7a6ff17669dfddf9"
server_key_iv: "8579da4ebe3dddfba6a5f0b309786a85"
server_key_version: "encryptionkey@0"
I20260926 20:15:44.785059 387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:44.799036 387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:44.802983 387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:44.803222 387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:44.803886 387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:15:44.804798 387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:44.804872 387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:44.804956 387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:44.805007 387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:44.815410 387 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:40001
I20260926 20:15:44.815444 515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:40001 every 8 connection(s)
I20260926 20:15:44.816489 387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:15:44.816931 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 387
I20260926 20:15:44.817065 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:15:44.817386 32223 external_mini_cluster.cc:1428] Setting key e81476da9db46c015045db5c43f5f7d3
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
Sep 26 20:15:44 dist-test-slave-vnbh krb5kdc[32748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453744, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:44.867498 518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:44.818737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:53993 (local address 127.31.119.254:33799)
0926 20:15:44.819021 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:44.819024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:44.819953 (+ 929us) server_negotiation.cc:407] Connection header received
0926 20:15:44.821901 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:44.821904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:44.821954 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:44.822079 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:44.823392 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:44.824400 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:44.825914 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:44.826384 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:44.826516 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:44.862340 (+ 35824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:44.862344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:44.862346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:44.862382 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:44.864413 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:44.865223 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:44.865226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:44.865228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:44.865400 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:44.865884 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:44.865892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:44.865895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:44.866217 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:44.866339 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:44.867031 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 20:15:44.867217 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260926 20:15:44.868331 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:44.819287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:33799 (local address 127.31.119.193:53993)
0926 20:15:44.819771 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:44.819826 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:44.821622 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:44.822290 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:44.822298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:44.822701 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:44.823230 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:44.823245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:44.824571 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:44.824574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:44.825770 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:44.825781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:44.826558 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:44.826561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:44.826742 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:44.827494 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:44.827511 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:44.862033 (+ 34522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:44.864593 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:44.864600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:44.864618 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:44.865047 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:44.865510 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:44.865513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:44.865517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:44.865713 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:44.866344 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:44.866351 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:44.866718 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 20:15:44.866899 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 20:15:44.867261 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":140,"threads_started":1}
I20260926 20:15:44.870098 516 heartbeater.cc:347] Connected to a master server at 127.31.119.254:33799
I20260926 20:15:44.870419 516 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:44.871068 516 heartbeater.cc:511] Master 127.31.119.254:33799 requested a full tablet report, sending...
I20260926 20:15:44.872786 325 ts_manager.cc:194] Registered new tserver with Master: 645951120a0245c89b6f957378648e04 (127.31.119.193:40001)
I20260926 20:15:44.874063 325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:53993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
W20260926 20:15:45.333621 32223 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.119.194: real 0.514s user 0.001s sys 0.003s
I20260926 20:15:45.334192 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:33799
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:45.462075 522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:45.462383 522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:45.462430 522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:45.467202 522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:45.467286 522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:45.467368 522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:15:45.473803 522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:33799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:45.475035 522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:45.476146 522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:45.487309 530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:45.487323 527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:45.487370 522 server_base.cc:1034] running on GCE node
W20260926 20:15:45.487327 528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:45.488021 522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:45.488816 522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:45.490011 522 hybrid_clock.cc:648] HybridClock initialized: now 1790453745489978 us; error 66 us; skew 500 ppm
Sep 26 20:15:45 dist-test-slave-vnbh krb5kdc[32748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453745, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:45.493160 522 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:15:45.494201 522 webserver.cc:460] Webserver started at http://127.31.119.194:42267/ using document root <none> and password file <none>
I20260926 20:15:45.494808 522 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:45.494874 522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:45.495059 522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:45.497423 522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "8d1105cea7c849c084900904c4367729"
format_stamp: "Formatted at 2026-09-26 20:15:45 on dist-test-slave-vnbh"
server_key: "032026516730a47cf890be0df4900c53"
server_key_iv: "f06419b63a8151ed816bf743e43c21ea"
server_key_version: "encryptionkey@0"
I20260926 20:15:45.498039 522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "8d1105cea7c849c084900904c4367729"
format_stamp: "Formatted at 2026-09-26 20:15:45 on dist-test-slave-vnbh"
server_key: "032026516730a47cf890be0df4900c53"
server_key_iv: "f06419b63a8151ed816bf743e43c21ea"
server_key_version: "encryptionkey@0"
I20260926 20:15:45.502303 522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:15:45.505424 537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:45.507083 522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:15:45.507270 522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "8d1105cea7c849c084900904c4367729"
format_stamp: "Formatted at 2026-09-26 20:15:45 on dist-test-slave-vnbh"
server_key: "032026516730a47cf890be0df4900c53"
server_key_iv: "f06419b63a8151ed816bf743e43c21ea"
server_key_version: "encryptionkey@0"
I20260926 20:15:45.507406 522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:45.524245 522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:45.527956 522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:45.528187 522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:45.528836 522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:15:45.529783 522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:45.529830 522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:45.529872 522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:45.529887 522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:45.540033 522 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:38961
I20260926 20:15:45.540056 650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:38961 every 8 connection(s)
I20260926 20:15:45.541182 522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:15:45.542219 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 522
I20260926 20:15:45.542351 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:15:45.542650 32223 external_mini_cluster.cc:1428] Setting key 290a0c7b4d1a8e56d2ba9427deba2679
Sep 26 20:15:45 dist-test-slave-vnbh krb5kdc[32748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453745, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
I20260926 20:15:45.556764 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:45.543422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:60199 (local address 127.31.119.254:33799)
0926 20:15:45.543686 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:45.543690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:45.544741 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 20:15:45.546270 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:45.546273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:45.546330 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:45.546420 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:45.547697 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:45.548325 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:45.549570 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:45.549819 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:45.549927 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:45.552960 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:45.552963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:45.552965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:45.552991 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:45.554487 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:45.555063 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:45.555066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:45.555067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:45.555150 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:45.555498 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:45.555503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:45.555505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:45.555697 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:45.555813 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:45.556424 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 20:15:45.556561 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20260926 20:15:45.557778 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:45.544050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:33799 (local address 127.31.119.194:60199)
0926 20:15:45.544568 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:45.544627 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:45.546039 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:45.546608 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:45.546616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:45.547002 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:45.547554 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:45.547569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:45.548456 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:45.548460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:45.549455 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:45.549464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:45.549971 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:45.549975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:45.550161 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:45.550960 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:45.550977 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:45.552763 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:45.554620 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:45.554626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:45.554642 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:45.554938 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:45.555250 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:45.555254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:45.555256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:45.555403 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:45.555820 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:45.555829 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:45.556127 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 20:15:45.556373 (+ 246us) client_negotiation.cc:240] Negotiation successful
0926 20:15:45.556754 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":150,"threads_started":1}
I20260926 20:15:45.559088 651 heartbeater.cc:347] Connected to a master server at 127.31.119.254:33799
I20260926 20:15:45.559360 651 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:45.559904 651 heartbeater.cc:511] Master 127.31.119.254:33799 requested a full tablet report, sending...
I20260926 20:15:45.561126 325 ts_manager.cc:194] Registered new tserver with Master: 8d1105cea7c849c084900904c4367729 (127.31.119.194:38961)
I20260926 20:15:45.561751 325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:60199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
I20260926 20:15:45.876688 516 heartbeater.cc:503] Master 127.31.119.254:33799 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
W20260926 20:15:46.442909 32223 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.119.195: real 0.898s user 0.000s sys 0.003s
I20260926 20:15:46.443475 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:33799
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:46.562037 657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:46.562503 657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:46.562575 657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 20:15:46.564179 651 heartbeater.cc:503] Master 127.31.119.254:33799 was elected leader, sending a full tablet report...
W20260926 20:15:46.568569 657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:46.568645 657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:46.568713 657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:15:46.574762 657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:33799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:46.575893 657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:46.576925 657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:46.587697 662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:46.587690 663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:46.587888 657 server_base.cc:1034] running on GCE node
W20260926 20:15:46.588107 665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:46.588593 657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:46.589359 657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:46.590559 657 hybrid_clock.cc:648] HybridClock initialized: now 1790453746590499 us; error 71 us; skew 500 ppm
Sep 26 20:15:46 dist-test-slave-vnbh krb5kdc[32748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453746, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:46.593726 657 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:15:46.594786 657 webserver.cc:460] Webserver started at http://127.31.119.195:45411/ using document root <none> and password file <none>
I20260926 20:15:46.595423 657 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:46.595520 657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:46.595755 657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:46.598030 657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "ea54b02b4cf8459b9356092033b281f3"
format_stamp: "Formatted at 2026-09-26 20:15:46 on dist-test-slave-vnbh"
server_key: "b9a1b1e1fcba9e8fbc45d9565dadf2b5"
server_key_iv: "db91b7f56647f4999c106484a1c89f12"
server_key_version: "encryptionkey@0"
I20260926 20:15:46.598647 657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "ea54b02b4cf8459b9356092033b281f3"
format_stamp: "Formatted at 2026-09-26 20:15:46 on dist-test-slave-vnbh"
server_key: "b9a1b1e1fcba9e8fbc45d9565dadf2b5"
server_key_iv: "db91b7f56647f4999c106484a1c89f12"
server_key_version: "encryptionkey@0"
I20260926 20:15:46.602809 657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:15:46.605420 672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:46.606928 657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:15:46.607080 657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "ea54b02b4cf8459b9356092033b281f3"
format_stamp: "Formatted at 2026-09-26 20:15:46 on dist-test-slave-vnbh"
server_key: "b9a1b1e1fcba9e8fbc45d9565dadf2b5"
server_key_iv: "db91b7f56647f4999c106484a1c89f12"
server_key_version: "encryptionkey@0"
I20260926 20:15:46.607208 657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:46.619908 657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:46.623440 657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:46.623660 657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:46.624293 657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:15:46.625207 657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:46.625331 657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:46.625419 657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:46.625463 657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:46.634953 657 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:44789
I20260926 20:15:46.634969 785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:44789 every 8 connection(s)
I20260926 20:15:46.636004 657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:15:46.640359 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 657
I20260926 20:15:46.640508 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:15:46.640767 32223 external_mini_cluster.cc:1428] Setting key 938b9bcbd690b4a5966ff37c7787d89f
Sep 26 20:15:46 dist-test-slave-vnbh krb5kdc[32748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453746, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:46.650969 788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.638188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:51469 (local address 127.31.119.254:33799)
0926 20:15:46.638444 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:46.638448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:46.639111 (+ 663us) server_negotiation.cc:407] Connection header received
0926 20:15:46.640519 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:46.640523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:46.640584 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:46.640682 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:46.642341 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.643119 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.644288 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.644528 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.644624 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.647329 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:46.647333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:46.647335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:46.647359 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:46.648873 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:46.649473 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:46.649476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:46.649477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:46.649559 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:46.649900 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:46.649902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:46.649904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:46.650087 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:46.650164 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:46.650700 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 20:15:46.650814 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":103,"threads_started":1}
I20260926 20:15:46.651845 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.638495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:33799 (local address 127.31.119.195:51469)
0926 20:15:46.638967 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:46.639011 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:46.640300 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:46.640972 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:46.640979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:46.641476 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:46.642208 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.642230 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.643244 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.643247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.644150 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.644160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.644765 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.644768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.644923 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.645591 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:46.645608 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:46.647105 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:46.648976 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:46.648981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:46.648995 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:46.649377 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:46.649639 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:46.649641 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:46.649642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:46.649808 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:46.650171 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:46.650176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:46.650433 (+ 257us) client_negotiation.cc:769] Sending connection context
0926 20:15:46.650594 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 20:15:46.650897 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260926 20:15:46.653129 786 heartbeater.cc:347] Connected to a master server at 127.31.119.254:33799
I20260926 20:15:46.653506 786 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:46.654175 786 heartbeater.cc:511] Master 127.31.119.254:33799 requested a full tablet report, sending...
I20260926 20:15:46.655454 325 ts_manager.cc:194] Registered new tserver with Master: ea54b02b4cf8459b9356092033b281f3 (127.31.119.195:44789)
I20260926 20:15:46.656122 325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:51469
I20260926 20:15:46.667649 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:15:46.680145 788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.671754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.31.119.254:33799)
0926 20:15:46.671891 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:46.671895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:46.671910 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:15:46.672054 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:46.672057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:46.672108 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:46.672189 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:46.673043 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.673687 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.674790 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.675018 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.675109 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.676544 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:46.676547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:46.676550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:46.676584 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:46.678358 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:46.678843 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:46.678846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:46.678847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:46.678934 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:46.679225 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:46.679229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:46.679231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:46.679484 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:46.679599 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:46.679906 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 20:15:46.680020 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:15:46.686074 325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:15:46.689384 325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:15:46.705075 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.699181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40001 (local address 127.0.0.1:54774)
0926 20:15:46.699732 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:46.699773 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:46.700042 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:46.700743 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:46.700751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:46.700780 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:46.700929 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.700935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.702117 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.702123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.703432 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.703441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.704513 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.704517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.704620 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.704655 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:15:46.704772 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:15:46.704899 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":120,"threads_started":1}
I20260926 20:15:46.705318 801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.699145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.31.119.195:44789)
0926 20:15:46.700032 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:46.700045 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:46.700069 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:15:46.700246 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:46.700253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:46.700475 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:46.700637 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:46.701093 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.702314 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.703844 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.704655 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.704809 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.704956 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:46.705012 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 20:15:46.705118 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":140,"threads_started":1}
I20260926 20:15:46.705582 800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.698996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:44789 (local address 127.0.0.1:52706)
0926 20:15:46.699546 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:46.699997 (+ 451us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:46.700063 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:46.700841 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:46.700845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:46.700863 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:46.700962 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.700969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.702450 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.702453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.703734 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.703741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.704801 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.704804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.704890 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.704909 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:15:46.704977 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:15:46.705030 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":118,"threads_started":1}
I20260926 20:15:46.705582 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.699775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38961 (local address 127.0.0.1:57608)
0926 20:15:46.700702 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:46.700715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:46.700790 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:46.701448 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:46.701451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:46.701471 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:46.701549 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.701555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.702847 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.702853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.703927 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.703937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.705125 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.705129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.705225 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.705261 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:15:46.705360 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:15:46.705444 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":95,"threads_started":1}
I20260926 20:15:46.706070 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.700325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.31.119.194:38961)
0926 20:15:46.700957 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:46.700963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:46.700984 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:15:46.701058 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:46.701063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:46.701261 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:46.701447 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:46.701664 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.702718 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.704058 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.704849 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.704958 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.705064 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:46.705703 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 20:15:46.705950 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":117,"threads_started":1}
I20260926 20:15:46.706532 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.699690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.31.119.193:40001)
0926 20:15:46.700331 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:46.700338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:46.700359 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:15:46.700421 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:46.700425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:46.700582 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:46.700707 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:46.701046 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.702013 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.703610 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.704388 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.704525 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.704672 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:46.706268 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0926 20:15:46.706389 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":150,"threads_started":1}
I20260926 20:15:46.708963 450 tablet_service.cc:1467] Processing CreateTablet for tablet 45422bbf621c464fae7bbce7895df2b2 (DEFAULT_TABLE table=test-table [id=0710a60963854014bbf998785e4bc952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:46.708973 585 tablet_service.cc:1467] Processing CreateTablet for tablet 45422bbf621c464fae7bbce7895df2b2 (DEFAULT_TABLE table=test-table [id=0710a60963854014bbf998785e4bc952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:46.710150 450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45422bbf621c464fae7bbce7895df2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:46.710155 585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45422bbf621c464fae7bbce7895df2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:46.711812 720 tablet_service.cc:1467] Processing CreateTablet for tablet 45422bbf621c464fae7bbce7895df2b2 (DEFAULT_TABLE table=test-table [id=0710a60963854014bbf998785e4bc952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:46.712944 720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45422bbf621c464fae7bbce7895df2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:46.716945 805 tablet_bootstrap.cc:492] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729: Bootstrap starting.
I20260926 20:15:46.719797 805 tablet_bootstrap.cc:654] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:46.720685 805 log.cc:824] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:46.720796 807 tablet_bootstrap.cc:492] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3: Bootstrap starting.
I20260926 20:15:46.721403 806 tablet_bootstrap.cc:492] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04: Bootstrap starting.
I20260926 20:15:46.722990 807 tablet_bootstrap.cc:654] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:46.723168 805 tablet_bootstrap.cc:492] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729: No bootstrap required, opened a new log
I20260926 20:15:46.723388 805 ts_tablet_manager.cc:1397] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 20:15:46.723652 806 tablet_bootstrap.cc:654] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:46.724537 806 log.cc:824] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:46.724597 807 log.cc:824] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:46.726936 807 tablet_bootstrap.cc:492] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3: No bootstrap required, opened a new log
I20260926 20:15:46.727157 807 ts_tablet_manager.cc:1397] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:15:46.728067 805 raft_consensus.cc:348] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea54b02b4cf8459b9356092033b281f3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44789 } } peers { permanent_uuid: "645951120a0245c89b6f957378648e04" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40001 } } peers { permanent_uuid: "8d1105cea7c849c084900904c4367729" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38961 } }
I20260926 20:15:46.728371 805 raft_consensus.cc:374] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:46.728396 806 tablet_bootstrap.cc:492] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04: No bootstrap required, opened a new log
I20260926 20:15:46.728428 805 raft_consensus.cc:729] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1105cea7c849c084900904c4367729, State: Initialized, Role: FOLLOWER
I20260926 20:15:46.728581 806 ts_tablet_manager.cc:1397] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 20:15:46.728931 805 consensus_queue.cc:260] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea54b02b4cf8459b9356092033b281f3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44789 } } peers { permanent_uuid: "645951120a0245c89b6f957378648e04" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40001 } } peers { permanent_uuid: "8d1105cea7c849c084900904c4367729" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38961 } }
I20260926 20:15:46.730104 805 ts_tablet_manager.cc:1428] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 20:15:46.731076 807 raft_consensus.cc:348] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea54b02b4cf8459b9356092033b281f3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44789 } } peers { permanent_uuid: "645951120a0245c89b6f957378648e04" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40001 } } peers { permanent_uuid: "8d1105cea7c849c084900904c4367729" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38961 } }
I20260926 20:15:46.731390 807 raft_consensus.cc:374] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:46.731447 807 raft_consensus.cc:729] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea54b02b4cf8459b9356092033b281f3, State: Initialized, Role: FOLLOWER
I20260926 20:15:46.731916 807 consensus_queue.cc:260] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea54b02b4cf8459b9356092033b281f3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44789 } } peers { permanent_uuid: "645951120a0245c89b6f957378648e04" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40001 } } peers { permanent_uuid: "8d1105cea7c849c084900904c4367729" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38961 } }
I20260926 20:15:46.732867 807 ts_tablet_manager.cc:1428] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:15:46.733083 786 heartbeater.cc:503] Master 127.31.119.254:33799 was elected leader, sending a full tablet report...
I20260926 20:15:46.733652 806 raft_consensus.cc:348] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea54b02b4cf8459b9356092033b281f3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44789 } } peers { permanent_uuid: "645951120a0245c89b6f957378648e04" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40001 } } peers { permanent_uuid: "8d1105cea7c849c084900904c4367729" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38961 } }
I20260926 20:15:46.733978 806 raft_consensus.cc:374] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:46.734053 806 raft_consensus.cc:729] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645951120a0245c89b6f957378648e04, State: Initialized, Role: FOLLOWER
I20260926 20:15:46.734549 806 consensus_queue.cc:260] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea54b02b4cf8459b9356092033b281f3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44789 } } peers { permanent_uuid: "645951120a0245c89b6f957378648e04" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40001 } } peers { permanent_uuid: "8d1105cea7c849c084900904c4367729" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38961 } }
I20260926 20:15:46.735827 806 ts_tablet_manager.cc:1428] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260926 20:15:46.794215 652 tablet.cc:2369] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:15:46.812352 812 raft_consensus.cc:482] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:15:46.812615 812 raft_consensus.cc:504] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea54b02b4cf8459b9356092033b281f3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44789 } } peers { permanent_uuid: "645951120a0245c89b6f957378648e04" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40001 } } peers { permanent_uuid: "8d1105cea7c849c084900904c4367729" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38961 } }
I20260926 20:15:46.813869 812 leader_election.cc:288] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 645951120a0245c89b6f957378648e04 (127.31.119.193:40001), 8d1105cea7c849c084900904c4367729 (127.31.119.194:38961)
I20260926 20:15:46.818313 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.814632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:42209 (local address 127.31.119.194:38961)
0926 20:15:46.814868 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:46.814873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:46.815006 (+ 133us) server_negotiation.cc:407] Connection header received
0926 20:15:46.815129 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:46.815132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:46.815179 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:46.815249 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:46.815554 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.816117 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.817198 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.817678 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.817813 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.817868 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:46.818068 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 20:15:46.818150 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260926 20:15:46.818441 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.814145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40001 (local address 127.31.119.195:54039)
0926 20:15:46.814275 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:46.814291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:46.814398 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:46.814680 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:46.814684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:46.814705 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:46.814848 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.814855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.815892 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.815895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.817121 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.817131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.818035 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.818042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.818141 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.818162 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:15:46.818236 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:15:46.818303 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1}
I20260926 20:15:46.818452 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.814266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:54039 (local address 127.31.119.193:40001)
0926 20:15:46.814394 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:46.814397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:46.814413 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:15:46.814492 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:46.814495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:46.814551 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:46.814664 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:46.814957 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.815725 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.817274 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.817909 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.818031 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.818093 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:46.818259 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:15:46.818332 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:15:46.818694 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.814521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38961 (local address 127.31.119.195:42209)
0926 20:15:46.814933 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:46.814944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:46.815038 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:46.815267 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:46.815268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:46.815282 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:46.815435 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.815441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.816227 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.816233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.817068 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:46.817077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.817798 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.817801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:46.817903 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.817926 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:15:46.818461 (+ 535us) client_negotiation.cc:240] Negotiation successful
0926 20:15:46.818538 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":23,"thread_start_us":124,"threads_started":1}
W20260926 20:15:46.819423 517 tablet.cc:2369] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:15:46.819562 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45422bbf621c464fae7bbce7895df2b2" candidate_uuid: "ea54b02b4cf8459b9356092033b281f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d1105cea7c849c084900904c4367729" is_pre_election: true
I20260926 20:15:46.819562 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45422bbf621c464fae7bbce7895df2b2" candidate_uuid: "ea54b02b4cf8459b9356092033b281f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "645951120a0245c89b6f957378648e04" is_pre_election: true
I20260926 20:15:46.819876 605 raft_consensus.cc:2454] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea54b02b4cf8459b9356092033b281f3 in term 0.
I20260926 20:15:46.819876 470 raft_consensus.cc:2454] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea54b02b4cf8459b9356092033b281f3 in term 0.
I20260926 20:15:46.820382 675 leader_election.cc:302] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 645951120a0245c89b6f957378648e04, ea54b02b4cf8459b9356092033b281f3; no voters:
I20260926 20:15:46.820684 812 raft_consensus.cc:2781] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:15:46.820775 812 raft_consensus.cc:482] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:15:46.820853 812 raft_consensus.cc:3037] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:46.822255 812 raft_consensus.cc:504] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea54b02b4cf8459b9356092033b281f3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44789 } } peers { permanent_uuid: "645951120a0245c89b6f957378648e04" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40001 } } peers { permanent_uuid: "8d1105cea7c849c084900904c4367729" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38961 } }
I20260926 20:15:46.822676 812 leader_election.cc:288] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [CANDIDATE]: Term 1 election: Requested vote from peers 645951120a0245c89b6f957378648e04 (127.31.119.193:40001), 8d1105cea7c849c084900904c4367729 (127.31.119.194:38961)
I20260926 20:15:46.823143 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45422bbf621c464fae7bbce7895df2b2" candidate_uuid: "ea54b02b4cf8459b9356092033b281f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "645951120a0245c89b6f957378648e04"
I20260926 20:15:46.823295 470 raft_consensus.cc:3037] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:46.823381 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45422bbf621c464fae7bbce7895df2b2" candidate_uuid: "ea54b02b4cf8459b9356092033b281f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d1105cea7c849c084900904c4367729"
I20260926 20:15:46.823534 605 raft_consensus.cc:3037] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:46.824625 470 raft_consensus.cc:2454] T 45422bbf621c464fae7bbce7895df2b2 P 645951120a0245c89b6f957378648e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea54b02b4cf8459b9356092033b281f3 in term 1.
I20260926 20:15:46.824899 605 raft_consensus.cc:2454] T 45422bbf621c464fae7bbce7895df2b2 P 8d1105cea7c849c084900904c4367729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea54b02b4cf8459b9356092033b281f3 in term 1.
I20260926 20:15:46.824970 675 leader_election.cc:302] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 645951120a0245c89b6f957378648e04, ea54b02b4cf8459b9356092033b281f3; no voters:
I20260926 20:15:46.825174 812 raft_consensus.cc:2781] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:46.825441 812 raft_consensus.cc:686] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [term 1 LEADER]: Becoming Leader. State: Replica: ea54b02b4cf8459b9356092033b281f3, State: Running, Role: LEADER
I20260926 20:15:46.825842 812 consensus_queue.cc:237] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea54b02b4cf8459b9356092033b281f3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44789 } } peers { permanent_uuid: "645951120a0245c89b6f957378648e04" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40001 } } peers { permanent_uuid: "8d1105cea7c849c084900904c4367729" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38961 } }
I20260926 20:15:46.829160 324 catalog_manager.cc:5495] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea54b02b4cf8459b9356092033b281f3 (127.31.119.195). New cstate: current_term: 1 leader_uuid: "ea54b02b4cf8459b9356092033b281f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea54b02b4cf8459b9356092033b281f3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "645951120a0245c89b6f957378648e04" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d1105cea7c849c084900904c4367729" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38961 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:15:46 dist-test-slave-vnbh krb5kdc[32748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:15:46.866521 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.856685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.31.119.193:40001)
0926 20:15:46.856866 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:46.856871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:46.857143 (+ 272us) server_negotiation.cc:407] Connection header received
0926 20:15:46.857356 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:46.857359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:46.857406 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:46.857500 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:46.858286 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.858809 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.859682 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.859904 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.860013 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.862760 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:46.862781 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:46.862796 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:46.862827 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:46.864578 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:46.865074 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:46.865080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:46.865081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:46.865166 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:46.865524 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:46.865527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:46.865529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:46.865819 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:46.866073 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:46.866216 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 20:15:46.866348 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 20:15:46 dist-test-slave-vnbh krb5kdc[32748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260926 20:15:46.887005 787 tablet.cc:2369] T 45422bbf621c464fae7bbce7895df2b2 P ea54b02b4cf8459b9356092033b281f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:15:46 dist-test-slave-vnbh krb5kdc[32748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:15:46.895283 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.885714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.31.119.193:40001)
0926 20:15:46.885920 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:46.885925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:46.886042 (+ 117us) server_negotiation.cc:407] Connection header received
0926 20:15:46.886226 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:46.886229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:46.886275 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:46.886360 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:46.887072 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.887661 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.888509 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.888779 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.888899 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.891640 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:46.891644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:46.891646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:46.891675 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:46.893182 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:46.893845 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:46.893848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:46.893849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:46.893931 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:46.894263 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:46.894267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:46.894268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:46.894467 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:46.894583 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:46.894960 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 20:15:46.895077 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 20:15:46 dist-test-slave-vnbh krb5kdc[32748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:15:46 dist-test-slave-vnbh krb5kdc[32748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:15:46.922611 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:46.913796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.31.119.193:40001)
0926 20:15:46.913948 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:46.913954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:46.914153 (+ 199us) server_negotiation.cc:407] Connection header received
0926 20:15:46.914327 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:46.914330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:46.914384 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:46.914471 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:46.915250 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.915766 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.916635 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:46.916862 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:46.916973 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:46.919542 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:46.919546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:46.919548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:46.919578 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:46.920881 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:46.921369 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:46.921374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:46.921379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:46.921464 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:46.921744 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:46.921747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:46.921749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:46.921928 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:46.922096 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:46.922312 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 20:15:46.922441 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 20:15:46.923969 430 tablet_service.cc:2908] Rejecting scan request for tablet 45422bbf621c464fae7bbce7895df2b2: Uninitialized: clean time has not yet been initialized
I20260926 20:15:46.926148 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 387
I20260926 20:15:46.932291 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 522
I20260926 20:15:46.938371 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 657
I20260926 20:15:46.944182 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 32761
2026-09-26T20:15:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (6655 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:15:47 dist-test-slave-vnbh krb5kdc[829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:15:47 dist-test-slave-vnbh krb5kdc[829](info): set up 2 sockets
Sep 26 20:15:47 dist-test-slave-vnbh krb5kdc[829](info): commencing operation
krb5kdc: starting...
W20260926 20:15:47.998004 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:15:48 dist-test-slave-vnbh krb5kdc[829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:15:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:15:48.129279 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:40921
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:36169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:40921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:48.245023 842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:48.245354 842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:48.245450 842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:48.250319 842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:48.250396 842 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:15:48.250414 842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:48.250433 842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:48.250450 842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:48.250466 842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:48.256140 842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:36169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:40921
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:40921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:48.257256 842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:48.258375 842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:15:48.267448 842 server_base.cc:1034] running on GCE node
W20260926 20:15:48.267441 848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:48.267630 850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:48.267441 847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:48.268375 842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:48.269655 842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:48.270860 842 hybrid_clock.cc:648] HybridClock initialized: now 1790453748270825 us; error 50 us; skew 500 ppm
Sep 26 20:15:48 dist-test-slave-vnbh krb5kdc[829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453748, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:48.281137 842 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:15:48.282388 842 webserver.cc:460] Webserver started at http://127.31.119.254:34969/ using document root <none> and password file <none>
I20260926 20:15:48.283030 842 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:48.283133 842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:48.283387 842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:48.285732 842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "68a35a9f67124c5998055f499417216b"
format_stamp: "Formatted at 2026-09-26 20:15:48 on dist-test-slave-vnbh"
server_key: "453c886dcaee9b40d632f52783155cf0"
server_key_iv: "e3825c2f1cbdfd2ac03f46fd211365aa"
server_key_version: "encryptionkey@0"
I20260926 20:15:48.286341 842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "68a35a9f67124c5998055f499417216b"
format_stamp: "Formatted at 2026-09-26 20:15:48 on dist-test-slave-vnbh"
server_key: "453c886dcaee9b40d632f52783155cf0"
server_key_iv: "e3825c2f1cbdfd2ac03f46fd211365aa"
server_key_version: "encryptionkey@0"
I20260926 20:15:48.290385 842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:15:48.293030 857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:48.294471 842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:15:48.294641 842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "68a35a9f67124c5998055f499417216b"
format_stamp: "Formatted at 2026-09-26 20:15:48 on dist-test-slave-vnbh"
server_key: "453c886dcaee9b40d632f52783155cf0"
server_key_iv: "e3825c2f1cbdfd2ac03f46fd211365aa"
server_key_version: "encryptionkey@0"
I20260926 20:15:48.294776 842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:48.304673 842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:48.308343 842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:48.308574 842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:48.317375 842 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:40921
I20260926 20:15:48.317436 919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:40921 every 8 connection(s)
I20260926 20:15:48.318444 842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:15:48.321667 920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:48.326344 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 842
I20260926 20:15:48.326484 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:15:48.326802 32223 external_mini_cluster.cc:1428] Setting key 6f16a247e0c4b16afc18df0da93f76da
I20260926 20:15:48.329088 920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b: Bootstrap starting.
I20260926 20:15:48.331614 920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:48.332465 920 log.cc:824] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:48.334733 920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b: No bootstrap required, opened a new log
Sep 26 20:15:48 dist-test-slave-vnbh krb5kdc[829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:48.338981 920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a35a9f67124c5998055f499417216b" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40921 } }
I20260926 20:15:48.339287 920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:48.339375 920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a35a9f67124c5998055f499417216b, State: Initialized, Role: FOLLOWER
I20260926 20:15:48.339932 920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a35a9f67124c5998055f499417216b" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40921 } }
I20260926 20:15:48.340109 920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:15:48.340193 920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:15:48.340329 920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:48.341559 920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a35a9f67124c5998055f499417216b" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40921 } }
I20260926 20:15:48.341943 920 leader_election.cc:302] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68a35a9f67124c5998055f499417216b; no voters:
I20260926 20:15:48.342105 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:48.328436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.31.119.254:40921)
0926 20:15:48.328948 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:48.328963 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:48.329002 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:15:48.329773 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:48.329794 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:48.330145 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:48.330497 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:48.331360 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.332453 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.333327 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.333831 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.334001 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:48.336612 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:48.336617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:48.336632 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:48.336663 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:48.338628 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:48.339165 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:48.339171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:48.339175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:48.339316 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:48.339700 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:48.339703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:48.339705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:48.340204 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:48.340413 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:48.340843 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 20:15:48.341166 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":175,"threads_started":1}
I20260926 20:15:48.342370 920 leader_election.cc:288] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:15:48.342747 926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:48.342948 926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [term 1 LEADER]: Becoming Leader. State: Replica: 68a35a9f67124c5998055f499417216b, State: Running, Role: LEADER
I20260926 20:15:48.343321 926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a35a9f67124c5998055f499417216b" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40921 } }
I20260926 20:15:48.343775 920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:48.345930 925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68a35a9f67124c5998055f499417216b. Latest consensus state: current_term: 1 leader_uuid: "68a35a9f67124c5998055f499417216b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a35a9f67124c5998055f499417216b" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40921 } } }
I20260926 20:15:48.346076 925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:48.345937 927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68a35a9f67124c5998055f499417216b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a35a9f67124c5998055f499417216b" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40921 } } }
I20260926 20:15:48.346410 927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b [sys.catalog]: This master's current role is: LEADER
W20260926 20:15:48.349522 941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:15:48.349614 941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:15:48.349691 940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:15:48.350558 940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:15:48.357699 940 catalog_manager.cc:1347] Generated new cluster ID: 32a6365c1af94829a91735ad08f476ae
I20260926 20:15:48.357813 940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:15:48.393414 940 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:15:48.394552 940 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:15:48.402899 940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b: Generated new TSK 0
I20260926 20:15:48.403709 940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:15:48.449505 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40921
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:36169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:48.592114 946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:48.592422 946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:48.592466 946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:48.596560 946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:48.596635 946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:48.596680 946 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:15:48.596715 946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:15:48.602370 946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:36169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.119.254:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:48.603487 946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:48.604521 946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:48.614817 952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:48.614868 954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:48.614859 951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:48.614971 946 server_base.cc:1034] running on GCE node
I20260926 20:15:48.615641 946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:48.616313 946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:48.617491 946 hybrid_clock.cc:648] HybridClock initialized: now 1790453748617450 us; error 53 us; skew 500 ppm
Sep 26 20:15:48 dist-test-slave-vnbh krb5kdc[829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453748, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:48.620507 946 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:15:48.621613 946 webserver.cc:460] Webserver started at http://127.31.119.193:43895/ using document root <none> and password file <none>
I20260926 20:15:48.622212 946 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:48.622301 946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:48.622531 946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:48.624686 946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "8a9ca210023e4f43bb53c1da57b7a5bb"
format_stamp: "Formatted at 2026-09-26 20:15:48 on dist-test-slave-vnbh"
server_key: "3e1948c31002ce0fba95cb5d02acedd8"
server_key_iv: "0cb098987dcfe1c2864ec3f9f937e623"
server_key_version: "encryptionkey@0"
I20260926 20:15:48.625365 946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "8a9ca210023e4f43bb53c1da57b7a5bb"
format_stamp: "Formatted at 2026-09-26 20:15:48 on dist-test-slave-vnbh"
server_key: "3e1948c31002ce0fba95cb5d02acedd8"
server_key_iv: "0cb098987dcfe1c2864ec3f9f937e623"
server_key_version: "encryptionkey@0"
I20260926 20:15:48.629558 946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:15:48.632313 961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:48.634148 946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:15:48.634292 946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "8a9ca210023e4f43bb53c1da57b7a5bb"
format_stamp: "Formatted at 2026-09-26 20:15:48 on dist-test-slave-vnbh"
server_key: "3e1948c31002ce0fba95cb5d02acedd8"
server_key_iv: "0cb098987dcfe1c2864ec3f9f937e623"
server_key_version: "encryptionkey@0"
I20260926 20:15:48.634421 946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:48.661012 946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:48.664698 946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:48.664912 946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:48.666637 946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:48.666711 946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:48.666787 946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:48.666826 946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:48.677835 946 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:42195
I20260926 20:15:48.677853 1076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:42195 every 8 connection(s)
Sep 26 20:15:48 dist-test-slave-vnbh krb5kdc[829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453748, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:48.678838 946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:15:48.682860 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:48.667556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:50119 (local address 127.31.119.254:40921)
0926 20:15:48.667778 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:48.667783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:48.668825 (+ 1042us) server_negotiation.cc:407] Connection header received
0926 20:15:48.670003 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:48.670009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:48.670078 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:48.670470 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:48.672383 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.673349 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.675343 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.675586 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.675672 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:48.678926 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:48.678931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:48.678934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:48.678965 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:48.680628 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:48.681210 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:48.681214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:48.681216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:48.681354 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:48.681692 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:48.681695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:48.681697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:48.681911 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:48.682018 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:48.682541 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 20:15:48.682678 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260926 20:15:48.683683 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:48.668095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40921 (local address 127.31.119.193:50119)
0926 20:15:48.668610 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:48.668656 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:48.669756 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:48.670809 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:48.670818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:48.671364 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:48.672167 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:48.672193 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.673528 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.673532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:48.675180 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:48.675193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.675759 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.675767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:48.676019 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:48.676819 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:48.676839 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:48.678729 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:48.680765 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:48.680771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:48.680787 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:48.681059 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:48.681449 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:48.681453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:48.681455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:48.681593 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:48.682016 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:48.682023 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:48.682297 (+ 274us) client_negotiation.cc:769] Sending connection context
0926 20:15:48.682497 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 20:15:48.682832 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":143,"threads_started":1}
I20260926 20:15:48.685058 1077 heartbeater.cc:347] Connected to a master server at 127.31.119.254:40921
I20260926 20:15:48.685221 1077 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:48.685784 1077 heartbeater.cc:511] Master 127.31.119.254:40921 requested a full tablet report, sending...
I20260926 20:15:48.687346 873 ts_manager.cc:194] Registered new tserver with Master: 8a9ca210023e4f43bb53c1da57b7a5bb (127.31.119.193:42195)
I20260926 20:15:48.687578 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 946
I20260926 20:15:48.687673 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:15:48.687945 32223 external_mini_cluster.cc:1428] Setting key 143362e93a28e42590bfe1772886c7f2
I20260926 20:15:48.688846 873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:50119
I20260926 20:15:48.695433 1083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:48.687441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40921 (local address 127.31.119.193:60393)
0926 20:15:48.687771 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:48.687783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:48.687893 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:48.688117 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:48.688120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:48.688361 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:48.688504 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:48.688512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.689371 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.689374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:48.690266 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:48.690281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.690847 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.690854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:48.690973 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:48.691547 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:48.691567 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:48.691911 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:48.693927 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:48.693930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:48.693932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:48.694148 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:48.694457 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:48.694460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:48.694461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:48.694537 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:48.694995 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:48.694997 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:48.695074 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 20:15:48.695159 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:15:48.695273 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
I20260926 20:15:48.695648 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:48.687495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:60393 (local address 127.31.119.254:40921)
0926 20:15:48.687622 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:48.687626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:48.687851 (+ 225us) server_negotiation.cc:407] Connection header received
0926 20:15:48.687971 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:48.687974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:48.688021 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:48.688101 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:48.688621 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.689212 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.690405 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.690713 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.690829 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:48.692051 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:48.692055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:48.692057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:48.692081 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:48.693792 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:48.694245 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:48.694251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:48.694255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:48.694354 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:48.694626 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:48.694630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:48.694632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:48.694899 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:48.695015 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:48.695250 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 20:15:48.695418 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:15:48.720414 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40921
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:36169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:48.839082 1086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:48.839411 1086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:48.839498 1086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:48.843779 1086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:48.843848 1086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:48.843890 1086 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:15:48.843982 1086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:15:48.850168 1086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:36169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.119.254:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:48.851316 1086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:48.852385 1086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:48.862628 1091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:48.862619 1092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:48.862619 1094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:48.862756 1086 server_base.cc:1034] running on GCE node
I20260926 20:15:48.863307 1086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:48.863921 1086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:48.865108 1086 hybrid_clock.cc:648] HybridClock initialized: now 1790453748865098 us; error 56 us; skew 500 ppm
Sep 26 20:15:48 dist-test-slave-vnbh krb5kdc[829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453748, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:48.867913 1086 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:15:48.868865 1086 webserver.cc:460] Webserver started at http://127.31.119.194:42089/ using document root <none> and password file <none>
I20260926 20:15:48.869503 1086 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:48.869596 1086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:48.869834 1086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:48.871941 1086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "fd02987a2167478d83002b643b15cd6b"
format_stamp: "Formatted at 2026-09-26 20:15:48 on dist-test-slave-vnbh"
server_key: "83de121ba930e7443c9126796f444170"
server_key_iv: "ffb7eebd19f19f461264cf7876fe4396"
server_key_version: "encryptionkey@0"
I20260926 20:15:48.872534 1086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "fd02987a2167478d83002b643b15cd6b"
format_stamp: "Formatted at 2026-09-26 20:15:48 on dist-test-slave-vnbh"
server_key: "83de121ba930e7443c9126796f444170"
server_key_iv: "ffb7eebd19f19f461264cf7876fe4396"
server_key_version: "encryptionkey@0"
I20260926 20:15:48.876468 1086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:15:48.879055 1101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:48.880604 1086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:48.880769 1086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "fd02987a2167478d83002b643b15cd6b"
format_stamp: "Formatted at 2026-09-26 20:15:48 on dist-test-slave-vnbh"
server_key: "83de121ba930e7443c9126796f444170"
server_key_iv: "ffb7eebd19f19f461264cf7876fe4396"
server_key_version: "encryptionkey@0"
I20260926 20:15:48.880897 1086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:48.895275 1086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:48.899471 1086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:48.899698 1086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:48.901475 1086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:48.901552 1086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:48.901633 1086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:48.901682 1086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:15:48 dist-test-slave-vnbh krb5kdc[829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453748, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:48.913154 1086 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:38581
I20260926 20:15:48.914223 1086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:15:48.913178 1216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:38581 every 8 connection(s)
I20260926 20:15:48.916654 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:48.902399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:56475 (local address 127.31.119.254:40921)
0926 20:15:48.902550 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:48.902555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:48.903649 (+ 1094us) server_negotiation.cc:407] Connection header received
0926 20:15:48.905038 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:48.905044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:48.905160 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:48.905232 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:48.907446 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.908003 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.909343 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.909577 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.909712 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:48.912423 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:48.912426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:48.912428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:48.912455 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:48.914023 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:48.914637 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:48.914644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:48.914646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:48.914794 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:48.915405 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:48.915409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:48.915410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:48.915616 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:48.915705 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:48.916406 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 20:15:48.916524 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:15:48.917485 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 1086
I20260926 20:15:48.917598 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:15:48.917717 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:48.902940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40921 (local address 127.31.119.194:56475)
0926 20:15:48.903443 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:48.903513 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:48.904733 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:48.905796 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:48.905809 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:48.906388 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:48.907234 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:48.907261 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.908153 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.908156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:48.909170 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:48.909179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.909771 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.909775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:48.909953 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:48.910600 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:48.910618 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:48.912231 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:48.914210 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:48.914217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:48.914233 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:48.914530 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:48.915097 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:48.915102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:48.915105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:48.915295 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:48.915918 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:48.915924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:48.916172 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 20:15:48.916345 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 20:15:48.916626 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260926 20:15:48.917871 32223 external_mini_cluster.cc:1428] Setting key a9f43831831acd6e16bb0c53456e6b5a
I20260926 20:15:48.919188 1217 heartbeater.cc:347] Connected to a master server at 127.31.119.254:40921
I20260926 20:15:48.919353 1217 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:48.919997 1217 heartbeater.cc:511] Master 127.31.119.254:40921 requested a full tablet report, sending...
I20260926 20:15:48.921115 874 ts_manager.cc:194] Registered new tserver with Master: fd02987a2167478d83002b643b15cd6b (127.31.119.194:38581)
I20260926 20:15:48.921867 874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:56475
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
I20260926 20:15:48.929353 1224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:48.921413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40921 (local address 127.31.119.194:52659)
0926 20:15:48.921698 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:48.921713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:48.921835 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:48.922097 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:48.922101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:48.922358 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:48.922496 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:48.922506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.923505 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.923507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:48.924064 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:48.924070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.924693 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.924695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:48.924827 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:48.925421 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:48.925432 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:48.925760 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:48.927718 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:48.927721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:48.927722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:48.927951 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:48.928237 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:48.928239 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:48.928240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:48.928311 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:48.928858 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:48.928860 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:48.928933 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 20:15:48.929014 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:15:48.929121 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":123,"threads_started":1}
I20260926 20:15:48.929445 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:48.921440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:52659 (local address 127.31.119.254:40921)
0926 20:15:48.921554 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:48.921558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:48.921802 (+ 244us) server_negotiation.cc:407] Connection header received
0926 20:15:48.921953 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:48.921957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:48.922000 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:48.922079 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:48.922639 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.923377 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.924170 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:48.924580 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:48.924709 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:48.925899 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:48.925906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:48.925911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:48.925945 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:48.927610 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:48.928050 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:48.928054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:48.928055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:48.928144 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:48.928419 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:48.928422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:48.928424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:48.928764 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:48.928872 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:48.929093 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 20:15:48.929228 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:15:48.949385 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40921
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:36169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:49.068559 1226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:49.068888 1226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:49.068936 1226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:49.073180 1226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:49.073272 1226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:49.073336 1226 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:15:49.073371 1226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:15:49.079105 1226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:36169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.119.254:40921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:49.080216 1226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:49.081228 1226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:49.091282 1232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:49.091284 1234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:49.091270 1231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:49.091364 1226 server_base.cc:1034] running on GCE node
I20260926 20:15:49.091971 1226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:49.092670 1226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:49.093876 1226 hybrid_clock.cc:648] HybridClock initialized: now 1790453749093859 us; error 41 us; skew 500 ppm
Sep 26 20:15:49 dist-test-slave-vnbh krb5kdc[829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453749, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:49.096748 1226 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:15:49.097711 1226 webserver.cc:460] Webserver started at http://127.31.119.195:36229/ using document root <none> and password file <none>
I20260926 20:15:49.098255 1226 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:49.098315 1226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:49.098493 1226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:49.100777 1226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "e737227cbe1648abb1c842d79eba7e47"
format_stamp: "Formatted at 2026-09-26 20:15:49 on dist-test-slave-vnbh"
server_key: "50add5ed00cf38259e007ec0e5caa98d"
server_key_iv: "0baf7b4db35482d9f280e3d6f8958362"
server_key_version: "encryptionkey@0"
I20260926 20:15:49.101366 1226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "e737227cbe1648abb1c842d79eba7e47"
format_stamp: "Formatted at 2026-09-26 20:15:49 on dist-test-slave-vnbh"
server_key: "50add5ed00cf38259e007ec0e5caa98d"
server_key_iv: "0baf7b4db35482d9f280e3d6f8958362"
server_key_version: "encryptionkey@0"
I20260926 20:15:49.105039 1226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:49.107595 1241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:49.108892 1226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 20:15:49.109016 1226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "e737227cbe1648abb1c842d79eba7e47"
format_stamp: "Formatted at 2026-09-26 20:15:49 on dist-test-slave-vnbh"
server_key: "50add5ed00cf38259e007ec0e5caa98d"
server_key_iv: "0baf7b4db35482d9f280e3d6f8958362"
server_key_version: "encryptionkey@0"
I20260926 20:15:49.109125 1226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:49.121191 1226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:49.124531 1226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:49.124689 1226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:49.126252 1226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:49.126325 1226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:49.126398 1226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:49.126441 1226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:15:49 dist-test-slave-vnbh krb5kdc[829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453749, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:49.137377 1226 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:34669
I20260926 20:15:49.137503 1356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:34669 every 8 connection(s)
I20260926 20:15:49.138504 1226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:15:49.142159 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.127262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:43043 (local address 127.31.119.254:40921)
0926 20:15:49.127388 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.127391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.128459 (+ 1068us) server_negotiation.cc:407] Connection header received
0926 20:15:49.129684 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.129688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.129749 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.129863 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:49.131557 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.132428 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.134352 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.134575 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.134678 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.138060 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:49.138063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:49.138066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:49.138112 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:49.139976 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.140528 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.140530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.140532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.140619 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.140947 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.140950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.140951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.141135 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:49.141212 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.141903 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.142018 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:15:49.143003 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.127807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40921 (local address 127.31.119.195:43043)
0926 20:15:49.128308 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.128351 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.129452 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.130081 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.130094 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.130593 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:49.131369 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.131394 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.132552 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.132559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.134175 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.134187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.134743 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.134748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.134998 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.135793 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:49.135820 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:49.137866 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:49.140126 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:49.140132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:49.140151 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:49.140417 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:49.140706 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:49.140709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:49.140710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:49.140856 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:49.141247 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:49.141253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:49.141597 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.141822 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.142101 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":137,"threads_started":1}
I20260926 20:15:49.144429 1357 heartbeater.cc:347] Connected to a master server at 127.31.119.254:40921
I20260926 20:15:49.144603 1357 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:49.145242 1357 heartbeater.cc:511] Master 127.31.119.254:40921 requested a full tablet report, sending...
I20260926 20:15:49.146375 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 1226
I20260926 20:15:49.146385 874 ts_manager.cc:194] Registered new tserver with Master: e737227cbe1648abb1c842d79eba7e47 (127.31.119.195:34669)
I20260926 20:15:49.146507 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:15:49.146893 32223 external_mini_cluster.cc:1428] Setting key 7a87ffc72ae5120fb42a54eacfe083a7
I20260926 20:15:49.147075 874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:43043
I20260926 20:15:49.149534 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:15:49.156368 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.147220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:60271 (local address 127.31.119.254:40921)
0926 20:15:49.147357 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.147361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.148153 (+ 792us) server_negotiation.cc:407] Connection header received
0926 20:15:49.148402 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.148406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.148494 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.148581 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:49.149092 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.149818 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.151053 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.151279 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.151361 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.152708 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:49.152712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:49.152714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:49.152737 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:49.154511 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.154998 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.155004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.155008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.155105 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.155439 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.155446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.155451 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.155662 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:49.155743 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.156109 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.156247 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:15:49.156526 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.147209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40921 (local address 127.31.119.195:60271)
0926 20:15:49.148023 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.148043 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.148180 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.148614 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.148617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.148849 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:49.148972 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.148979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.149947 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.149951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.150922 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.150937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.151393 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.151399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.151521 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.152138 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:49.152150 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:49.152495 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:49.154679 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:49.154684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:49.154686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:49.154895 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:49.155234 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:49.155236 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:49.155238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:49.155331 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:49.155769 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:49.155772 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:49.155893 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.156030 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.156219 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":597,"threads_started":1}
Sep 26 20:15:49 dist-test-slave-vnbh krb5kdc[829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:15:49 dist-test-slave-vnbh krb5kdc[829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:49.178200 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.169018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.31.119.254:40921)
0926 20:15:49.169152 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.169156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.169196 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:15:49.169238 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.169240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.169332 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.169421 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:49.170188 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.170724 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.171608 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.171831 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.171935 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.174608 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:49.174612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:49.174615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:49.174644 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:49.176365 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.176890 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.176894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.176896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.176983 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.177261 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.177264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.177266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.177495 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:49.177634 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.177899 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.178022 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:15:49.181623 874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:15:49.185487 874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:15:49.200380 1375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.194445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38581 (local address 127.0.0.1:47732)
0926 20:15:49.195342 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.195353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.195423 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.196086 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.196090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.196107 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:49.196217 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.196227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.197650 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.197655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.198622 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.198632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.199771 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.199775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.199894 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.199943 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.200056 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.200200 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":25,"thread_start_us":104,"threads_started":1}
I20260926 20:15:49.200380 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.194262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:34669 (local address 127.0.0.1:52312)
0926 20:15:49.195001 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.195032 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.195120 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.195855 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.195857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.195871 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:49.195930 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.195935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.197244 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.197248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.198937 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.198948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.200025 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.200032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.200118 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.200137 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.200189 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.200247 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":270,"thread_start_us":118,"threads_started":1}
I20260926 20:15:49.200469 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.194954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.31.119.195:34669)
0926 20:15:49.195374 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.195389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.195414 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:15:49.195487 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.195492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.195702 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.196068 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:49.196208 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.197134 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.199091 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.199860 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.200040 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.200162 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.200231 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.200317 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":120,"threads_started":1}
I20260926 20:15:49.200834 1374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.194085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:42195 (local address 127.0.0.1:51528)
0926 20:15:49.194481 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.194542 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.194671 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.195726 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.195732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.195758 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:49.195924 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.195931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.197469 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.197472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.198945 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.198951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.200421 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.200427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.200515 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.200546 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.200625 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.200713 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":68,"thread_start_us":103,"threads_started":1}
I20260926 20:15:49.200920 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.194914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.31.119.193:42195)
0926 20:15:49.195203 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.195211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.195237 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:15:49.195318 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.195325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.195557 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.195721 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:49.196012 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.197350 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.199083 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.199860 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.200025 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.200162 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.200679 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.200790 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260926 20:15:49.203893 995 tablet_service.cc:1467] Processing CreateTablet for tablet ede09ed4b04e4521973d97ed4bc5ccd1 (DEFAULT_TABLE table=test-table [id=e89373e69c3744598d2502f9d4b3f37d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:49.203905 1291 tablet_service.cc:1467] Processing CreateTablet for tablet ede09ed4b04e4521973d97ed4bc5ccd1 (DEFAULT_TABLE table=test-table [id=e89373e69c3744598d2502f9d4b3f37d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:49.205101 995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede09ed4b04e4521973d97ed4bc5ccd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:49.205109 1291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede09ed4b04e4521973d97ed4bc5ccd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:49.205467 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.195282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.31.119.194:38581)
0926 20:15:49.195673 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.195684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.195704 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:15:49.195773 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.195778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.195934 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.196364 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:49.196491 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.197549 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.198792 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.199621 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.201231 (+ 1610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.201394 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.201433 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.201505 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":97,"threads_started":1}
I20260926 20:15:49.208194 1151 tablet_service.cc:1467] Processing CreateTablet for tablet ede09ed4b04e4521973d97ed4bc5ccd1 (DEFAULT_TABLE table=test-table [id=e89373e69c3744598d2502f9d4b3f37d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:49.209154 1151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede09ed4b04e4521973d97ed4bc5ccd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:49.213195 1380 tablet_bootstrap.cc:492] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: Bootstrap starting.
I20260926 20:15:49.214599 1381 tablet_bootstrap.cc:492] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: Bootstrap starting.
I20260926 20:15:49.215936 1380 tablet_bootstrap.cc:654] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:49.216375 1381 tablet_bootstrap.cc:654] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:49.216845 1380 log.cc:824] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:49.217161 1381 log.cc:824] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:49.217203 1382 tablet_bootstrap.cc:492] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: Bootstrap starting.
I20260926 20:15:49.219437 1382 tablet_bootstrap.cc:654] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:49.220203 1380 tablet_bootstrap.cc:492] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: No bootstrap required, opened a new log
I20260926 20:15:49.220232 1382 log.cc:824] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:49.220440 1380 ts_tablet_manager.cc:1397] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 20:15:49.222379 1382 tablet_bootstrap.cc:492] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: No bootstrap required, opened a new log
I20260926 20:15:49.222400 1381 tablet_bootstrap.cc:492] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: No bootstrap required, opened a new log
I20260926 20:15:49.222589 1381 ts_tablet_manager.cc:1397] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 20:15:49.222589 1382 ts_tablet_manager.cc:1397] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:15:49.224545 1380 raft_consensus.cc:348] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } }
I20260926 20:15:49.225009 1380 raft_consensus.cc:374] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:49.225126 1380 raft_consensus.cc:729] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9ca210023e4f43bb53c1da57b7a5bb, State: Initialized, Role: FOLLOWER
I20260926 20:15:49.225777 1380 consensus_queue.cc:260] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } }
I20260926 20:15:49.226542 1077 heartbeater.cc:503] Master 127.31.119.254:40921 was elected leader, sending a full tablet report...
I20260926 20:15:49.226673 1381 raft_consensus.cc:348] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } }
I20260926 20:15:49.226939 1380 ts_tablet_manager.cc:1428] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:15:49.226953 1381 raft_consensus.cc:374] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:49.227008 1381 raft_consensus.cc:729] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e737227cbe1648abb1c842d79eba7e47, State: Initialized, Role: FOLLOWER
I20260926 20:15:49.227463 1381 consensus_queue.cc:260] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } }
I20260926 20:15:49.227864 1382 raft_consensus.cc:348] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } }
I20260926 20:15:49.228125 1382 raft_consensus.cc:374] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:49.228227 1382 raft_consensus.cc:729] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd02987a2167478d83002b643b15cd6b, State: Initialized, Role: FOLLOWER
I20260926 20:15:49.228569 1381 ts_tablet_manager.cc:1428] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:15:49.228650 1382 consensus_queue.cc:260] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } }
I20260926 20:15:49.228930 1357 heartbeater.cc:503] Master 127.31.119.254:40921 was elected leader, sending a full tablet report...
I20260926 20:15:49.229417 1217 heartbeater.cc:503] Master 127.31.119.254:40921 was elected leader, sending a full tablet report...
I20260926 20:15:49.229667 1382 ts_tablet_manager.cc:1428] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260926 20:15:49.389868 1358 tablet.cc:2369] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:15:49.392009 1386 raft_consensus.cc:482] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:15:49.392248 1386 raft_consensus.cc:504] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } }
I20260926 20:15:49.393558 1386 leader_election.cc:288] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e737227cbe1648abb1c842d79eba7e47 (127.31.119.195:34669), fd02987a2167478d83002b643b15cd6b (127.31.119.194:38581)
I20260926 20:15:49.399482 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.394020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:46167 (local address 127.31.119.195:34669)
0926 20:15:49.394157 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.394162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.394427 (+ 265us) server_negotiation.cc:407] Connection header received
0926 20:15:49.394581 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.394586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.394652 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.394733 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:49.395227 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.396015 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.397969 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.398625 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.398760 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.398814 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.399272 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.399348 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:15:49.399482 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.394188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:55341 (local address 127.31.119.194:38581)
0926 20:15:49.394298 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.394303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.394488 (+ 185us) server_negotiation.cc:407] Connection header received
0926 20:15:49.394695 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.394698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.394751 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.394820 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:49.395203 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.396069 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.397969 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.398635 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.398760 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.398814 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.399272 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.399348 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:15:49.399617 1389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.393993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38581 (local address 127.31.119.193:55341)
0926 20:15:49.394405 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.394421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.394531 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.394848 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.394851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.394885 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:49.395043 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.395051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.396181 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.396185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.397816 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.397828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.398780 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.398784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.399044 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.399151 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.399239 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.399423 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":288,"spinlock_wait_cycles":3584,"thread_start_us":130,"threads_started":1}
I20260926 20:15:49.399678 1390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.393786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:34669 (local address 127.31.119.193:46167)
0926 20:15:49.394310 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.394334 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.394477 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.394776 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.394778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.394822 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:49.395049 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.395055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.396181 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.396185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.397816 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.397828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.398780 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.398784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.399048 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.399154 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.399239 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.399418 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":373,"spinlock_wait_cycles":17664,"thread_start_us":120,"threads_started":1}
I20260926 20:15:49.400275 1171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede09ed4b04e4521973d97ed4bc5ccd1" candidate_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd02987a2167478d83002b643b15cd6b" is_pre_election: true
I20260926 20:15:49.400273 1311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede09ed4b04e4521973d97ed4bc5ccd1" candidate_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e737227cbe1648abb1c842d79eba7e47" is_pre_election: true
I20260926 20:15:49.400578 1171 raft_consensus.cc:2454] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9ca210023e4f43bb53c1da57b7a5bb in term 0.
I20260926 20:15:49.400596 1311 raft_consensus.cc:2454] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9ca210023e4f43bb53c1da57b7a5bb in term 0.
I20260926 20:15:49.401129 963 leader_election.cc:302] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9ca210023e4f43bb53c1da57b7a5bb, e737227cbe1648abb1c842d79eba7e47; no voters:
I20260926 20:15:49.401470 1386 raft_consensus.cc:2781] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:15:49.401553 1386 raft_consensus.cc:482] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:15:49.401656 1386 raft_consensus.cc:3037] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:49.402730 1386 raft_consensus.cc:504] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } }
I20260926 20:15:49.403115 1386 leader_election.cc:288] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [CANDIDATE]: Term 1 election: Requested vote from peers e737227cbe1648abb1c842d79eba7e47 (127.31.119.195:34669), fd02987a2167478d83002b643b15cd6b (127.31.119.194:38581)
I20260926 20:15:49.403502 1171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede09ed4b04e4521973d97ed4bc5ccd1" candidate_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd02987a2167478d83002b643b15cd6b"
I20260926 20:15:49.403502 1311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede09ed4b04e4521973d97ed4bc5ccd1" candidate_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e737227cbe1648abb1c842d79eba7e47"
I20260926 20:15:49.403636 1171 raft_consensus.cc:3037] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:49.403648 1311 raft_consensus.cc:3037] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:49.405040 1171 raft_consensus.cc:2454] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9ca210023e4f43bb53c1da57b7a5bb in term 1.
I20260926 20:15:49.405426 1311 raft_consensus.cc:2454] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9ca210023e4f43bb53c1da57b7a5bb in term 1.
I20260926 20:15:49.405424 964 leader_election.cc:302] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9ca210023e4f43bb53c1da57b7a5bb, fd02987a2167478d83002b643b15cd6b; no voters:
I20260926 20:15:49.405619 1386 raft_consensus.cc:2781] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:49.405853 1386 raft_consensus.cc:686] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 LEADER]: Becoming Leader. State: Replica: 8a9ca210023e4f43bb53c1da57b7a5bb, State: Running, Role: LEADER
I20260926 20:15:49.406229 1386 consensus_queue.cc:237] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } }
I20260926 20:15:49.409503 874 catalog_manager.cc:5495] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9ca210023e4f43bb53c1da57b7a5bb (127.31.119.193). New cstate: current_term: 1 leader_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } health_report { overall_health: UNKNOWN } } }
W20260926 20:15:49.415506 1218 tablet.cc:2369] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:15:49.430841 1078 tablet.cc:2369] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:15:49.445358 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.441214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.31.119.193:42195)
0926 20:15:49.441407 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.441412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.441427 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:15:49.441721 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.441724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.441781 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.441884 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:49.442156 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.442667 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.443637 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.443852 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.443958 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.444336 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:49.444863 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:49.444985 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.445093 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.445154 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 20:15:49.451412 1311 raft_consensus.cc:1261] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47 [term 1 FOLLOWER]: Refusing update from remote peer 8a9ca210023e4f43bb53c1da57b7a5bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:49.451404 1171 raft_consensus.cc:1261] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b [term 1 FOLLOWER]: Refusing update from remote peer 8a9ca210023e4f43bb53c1da57b7a5bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:49.452148 1391 consensus_queue.cc:1035] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:49.452333 1386 consensus_queue.cc:1035] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:49.460052 1395 mvcc.cc:205] Tried to move back new op lower bound from 7333698557746044928 to 7333698557570781184. Current Snapshot: MvccSnapshot[applied={T|T < 7333698557746044928}]
I20260926 20:15:49.460716 1394 mvcc.cc:205] Tried to move back new op lower bound from 7333698557746044928 to 7333698557570781184. Current Snapshot: MvccSnapshot[applied={T|T < 7333698557746044928}]
I20260926 20:15:49.475667 872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:49.475909 872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:49.478752 872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b: Sending DeleteTablet for 3 replicas of tablet ede09ed4b04e4521973d97ed4bc5ccd1
I20260926 20:15:49.479774 1291 tablet_service.cc:1514] Processing DeleteTablet for tablet ede09ed4b04e4521973d97ed4bc5ccd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:49 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:52312
I20260926 20:15:49.479800 995 tablet_service.cc:1514] Processing DeleteTablet for tablet ede09ed4b04e4521973d97ed4bc5ccd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:49 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:51528
I20260926 20:15:49.479811 1151 tablet_service.cc:1514] Processing DeleteTablet for tablet ede09ed4b04e4521973d97ed4bc5ccd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:49 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:47732
I20260926 20:15:49.480356 1404 tablet_replica.cc:330] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: stopping tablet replica
I20260926 20:15:49.480355 1405 tablet_replica.cc:330] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: stopping tablet replica
I20260926 20:15:49.480706 1404 raft_consensus.cc:2229] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:15:49.480706 1405 raft_consensus.cc:2229] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:49.481047 1405 raft_consensus.cc:2258] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:49.481128 1404 raft_consensus.cc:2258] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:49.481632 1403 tablet_replica.cc:330] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: stopping tablet replica
I20260926 20:15:49.481813 872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:15:49.482283 872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:15:49.482831 1405 ts_tablet_manager.cc:1905] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:49.486449 1405 ts_tablet_manager.cc:1918] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:15:49.486451 1404 ts_tablet_manager.cc:1905] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:49.486560 1405 log.cc:1196] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/ede09ed4b04e4521973d97ed4bc5ccd1
I20260926 20:15:49.486907 1405 ts_tablet_manager.cc:1939] T ede09ed4b04e4521973d97ed4bc5ccd1 P e737227cbe1648abb1c842d79eba7e47: Deleting consensus metadata
I20260926 20:15:49.488245 995 tablet_service.cc:1467] Processing CreateTablet for tablet 9df67288e4a845beb992cb58787aaf34 (DEFAULT_TABLE table=test-table [id=a2599555cb45475099ecca29fcd4d24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:49.488373 1403 raft_consensus.cc:2229] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:49.488507 995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df67288e4a845beb992cb58787aaf34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:49.488741 1403 raft_consensus.cc:2258] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:49.489367 1151 tablet_service.cc:1467] Processing CreateTablet for tablet 9df67288e4a845beb992cb58787aaf34 (DEFAULT_TABLE table=test-table [id=a2599555cb45475099ecca29fcd4d24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:49.489488 1404 ts_tablet_manager.cc:1918] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:15:49.489574 1404 log.cc:1196] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/ede09ed4b04e4521973d97ed4bc5ccd1
I20260926 20:15:49.489647 1151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df67288e4a845beb992cb58787aaf34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:49.489871 1404 ts_tablet_manager.cc:1939] T ede09ed4b04e4521973d97ed4bc5ccd1 P 8a9ca210023e4f43bb53c1da57b7a5bb: Deleting consensus metadata
I20260926 20:15:49.490828 1380 tablet_bootstrap.cc:492] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb: Bootstrap starting.
I20260926 20:15:49.491310 859 catalog_manager.cc:4841] TS e737227cbe1648abb1c842d79eba7e47 (127.31.119.195:34669): tablet ede09ed4b04e4521973d97ed4bc5ccd1 (table test-table [id=e89373e69c3744598d2502f9d4b3f37d]) successfully deleted
I20260926 20:15:49.491827 1380 tablet_bootstrap.cc:654] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:49.492292 1382 tablet_bootstrap.cc:492] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b: Bootstrap starting.
I20260926 20:15:49.492848 858 catalog_manager.cc:4841] TS 8a9ca210023e4f43bb53c1da57b7a5bb (127.31.119.193:42195): tablet ede09ed4b04e4521973d97ed4bc5ccd1 (table test-table [id=e89373e69c3744598d2502f9d4b3f37d]) successfully deleted
I20260926 20:15:49.493275 1382 tablet_bootstrap.cc:654] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:49.493242 1291 tablet_service.cc:1467] Processing CreateTablet for tablet 9df67288e4a845beb992cb58787aaf34 (DEFAULT_TABLE table=test-table [id=a2599555cb45475099ecca29fcd4d24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:49.493398 1403 ts_tablet_manager.cc:1905] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:49.493518 1291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df67288e4a845beb992cb58787aaf34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:49.494379 1382 tablet_bootstrap.cc:492] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b: No bootstrap required, opened a new log
I20260926 20:15:49.494460 1382 ts_tablet_manager.cc:1397] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:15:49.495257 1382 raft_consensus.cc:348] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.495376 1382 raft_consensus.cc:374] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:49.495415 1382 raft_consensus.cc:729] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd02987a2167478d83002b643b15cd6b, State: Initialized, Role: FOLLOWER
I20260926 20:15:49.495550 1382 consensus_queue.cc:260] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.495858 1382 ts_tablet_manager.cc:1428] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:15:49.496304 1381 tablet_bootstrap.cc:492] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47: Bootstrap starting.
I20260926 20:15:49.497080 1403 ts_tablet_manager.cc:1918] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:15:49.497208 1403 log.cc:1196] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/ede09ed4b04e4521973d97ed4bc5ccd1
I20260926 20:15:49.497334 1381 tablet_bootstrap.cc:654] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:49.497594 1403 ts_tablet_manager.cc:1939] T ede09ed4b04e4521973d97ed4bc5ccd1 P fd02987a2167478d83002b643b15cd6b: Deleting consensus metadata
I20260926 20:15:49.498968 860 catalog_manager.cc:4841] TS fd02987a2167478d83002b643b15cd6b (127.31.119.194:38581): tablet ede09ed4b04e4521973d97ed4bc5ccd1 (table test-table [id=e89373e69c3744598d2502f9d4b3f37d]) successfully deleted
I20260926 20:15:49.499086 1381 tablet_bootstrap.cc:492] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47: No bootstrap required, opened a new log
I20260926 20:15:49.499166 1381 ts_tablet_manager.cc:1397] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 20:15:49.499328 1380 tablet_bootstrap.cc:492] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb: No bootstrap required, opened a new log
I20260926 20:15:49.499425 1380 ts_tablet_manager.cc:1397] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260926 20:15:49.499663 1381 raft_consensus.cc:348] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.499785 1381 raft_consensus.cc:374] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:49.499826 1381 raft_consensus.cc:729] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e737227cbe1648abb1c842d79eba7e47, State: Initialized, Role: FOLLOWER
I20260926 20:15:49.499905 1380 raft_consensus.cc:348] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.499953 1381 consensus_queue.cc:260] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.500018 1380 raft_consensus.cc:374] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:49.500058 1380 raft_consensus.cc:729] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9ca210023e4f43bb53c1da57b7a5bb, State: Initialized, Role: FOLLOWER
I20260926 20:15:49.500174 1380 consensus_queue.cc:260] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.500227 1381 ts_tablet_manager.cc:1428] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:15:49.500444 1380 ts_tablet_manager.cc:1428] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:15:49.550618 1387 raft_consensus.cc:482] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:15:49.550787 1387 raft_consensus.cc:504] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.552076 1387 leader_election.cc:288] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9ca210023e4f43bb53c1da57b7a5bb (127.31.119.193:42195), fd02987a2167478d83002b643b15cd6b (127.31.119.194:38581)
I20260926 20:15:49.556260 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.552523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:35683 (local address 127.31.119.193:42195)
0926 20:15:49.552650 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.552655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.553145 (+ 490us) server_negotiation.cc:407] Connection header received
0926 20:15:49.553260 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.553264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.553360 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.553438 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:49.553694 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.554185 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.555227 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.555635 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.555771 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.555828 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.556012 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.556092 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:15:49.556695 1409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.552335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:42195 (local address 127.31.119.195:35683)
0926 20:15:49.553041 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.553054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.553156 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.553449 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.553452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.553468 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:49.553582 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.553587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.554295 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.554301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.555105 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.555114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.555776 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.555779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.555874 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.555896 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.556460 (+ 564us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.556536 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":330,"thread_start_us":110,"threads_started":1}
I20260926 20:15:49.557019 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.552335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38581 (local address 127.31.119.195:55937)
0926 20:15:49.552477 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.552494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.552600 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.553010 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.553013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.553032 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:49.553158 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.553164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.554255 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.554259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.555481 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.555491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.556632 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.556636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.556747 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.556769 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.556838 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.556905 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 20:15:49.557271 1019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df67288e4a845beb992cb58787aaf34" candidate_uuid: "e737227cbe1648abb1c842d79eba7e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" is_pre_election: true
I20260926 20:15:49.557417 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.552582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:55937 (local address 127.31.119.194:38581)
0926 20:15:49.552698 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.552702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.552775 (+ 73us) server_negotiation.cc:407] Connection header received
0926 20:15:49.552839 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.552842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.552896 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.552983 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:49.553312 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.554122 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.555623 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.556230 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.556345 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.556390 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.557209 (+ 819us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.557305 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:15:49.557559 1019 raft_consensus.cc:2454] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e737227cbe1648abb1c842d79eba7e47 in term 0.
I20260926 20:15:49.557770 1171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df67288e4a845beb992cb58787aaf34" candidate_uuid: "e737227cbe1648abb1c842d79eba7e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd02987a2167478d83002b643b15cd6b" is_pre_election: true
I20260926 20:15:49.557911 1171 raft_consensus.cc:2454] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e737227cbe1648abb1c842d79eba7e47 in term 0.
I20260926 20:15:49.557988 1242 leader_election.cc:302] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9ca210023e4f43bb53c1da57b7a5bb, e737227cbe1648abb1c842d79eba7e47; no voters:
I20260926 20:15:49.558279 1387 raft_consensus.cc:2781] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:15:49.558341 1387 raft_consensus.cc:482] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:15:49.558379 1387 raft_consensus.cc:3037] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:49.559368 1387 raft_consensus.cc:504] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.559751 1387 leader_election.cc:288] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9ca210023e4f43bb53c1da57b7a5bb (127.31.119.193:42195), fd02987a2167478d83002b643b15cd6b (127.31.119.194:38581)
I20260926 20:15:49.560086 1019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df67288e4a845beb992cb58787aaf34" candidate_uuid: "e737227cbe1648abb1c842d79eba7e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb"
I20260926 20:15:49.560221 1019 raft_consensus.cc:3037] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:49.560245 1171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df67288e4a845beb992cb58787aaf34" candidate_uuid: "e737227cbe1648abb1c842d79eba7e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd02987a2167478d83002b643b15cd6b"
I20260926 20:15:49.560359 1171 raft_consensus.cc:3037] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:49.561437 1019 raft_consensus.cc:2454] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e737227cbe1648abb1c842d79eba7e47 in term 1.
I20260926 20:15:49.561437 1171 raft_consensus.cc:2454] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e737227cbe1648abb1c842d79eba7e47 in term 1.
I20260926 20:15:49.561740 1242 leader_election.cc:302] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9ca210023e4f43bb53c1da57b7a5bb, e737227cbe1648abb1c842d79eba7e47; no voters:
I20260926 20:15:49.561903 1387 raft_consensus.cc:2781] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:49.562142 1387 raft_consensus.cc:686] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 1 LEADER]: Becoming Leader. State: Replica: e737227cbe1648abb1c842d79eba7e47, State: Running, Role: LEADER
I20260926 20:15:49.562408 1387 consensus_queue.cc:237] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.564829 872 catalog_manager.cc:5495] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 reported cstate change: term changed from 0 to 1, leader changed from <none> to e737227cbe1648abb1c842d79eba7e47 (127.31.119.195). New cstate: current_term: 1 leader_uuid: "e737227cbe1648abb1c842d79eba7e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } health_report { overall_health: HEALTHY } } }
Sep 26 20:15:49 dist-test-slave-vnbh krb5kdc[829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453748, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:49.600629 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.591226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.31.119.254:40921)
0926 20:15:49.591369 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.591373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.591556 (+ 183us) server_negotiation.cc:407] Connection header received
0926 20:15:49.591704 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.591707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.591748 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.591815 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:49.592701 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.593271 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.594441 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.594644 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.594719 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.597353 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:49.597356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:49.597358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:49.597380 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:49.598868 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.599349 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.599354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.599355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.599429 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.599723 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.599726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.599727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.599903 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:49.600003 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.600298 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.600499 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":59}
I20260926 20:15:49.600629 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.591131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40921 (local address 127.0.0.1:50176)
0926 20:15:49.591455 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.591471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.591605 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.591874 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.591878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.592427 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:49.592558 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.592567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.593426 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.593430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.594326 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.594337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.594765 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.594769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.594860 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.595489 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:49.595505 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:49.597165 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:49.598977 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:49.598990 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:49.598997 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:49.599250 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:49.599545 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:49.599548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:49.599550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:49.599626 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:49.600003 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:49.600012 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:49.600130 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.600253 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.600431 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":125,"threads_started":1}
I20260926 20:15:49.604851 872 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:50176:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:15:49.605649 872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:15:49.610497 995 tablet_service.cc:1467] Processing CreateTablet for tablet b4feea81f7d04d0087f0ae49219b4ead (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fb4bb4ac3af4f2fba392649a2b0ff8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:15:49.610497 1151 tablet_service.cc:1467] Processing CreateTablet for tablet b4feea81f7d04d0087f0ae49219b4ead (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fb4bb4ac3af4f2fba392649a2b0ff8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:15:49.610734 1291 tablet_service.cc:1467] Processing CreateTablet for tablet b4feea81f7d04d0087f0ae49219b4ead (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fb4bb4ac3af4f2fba392649a2b0ff8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:15:49.610905 995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4feea81f7d04d0087f0ae49219b4ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:49.610905 1151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4feea81f7d04d0087f0ae49219b4ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:49.611145 1291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4feea81f7d04d0087f0ae49219b4ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:49.613838 1380 tablet_bootstrap.cc:492] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: Bootstrap starting.
I20260926 20:15:49.614089 1382 tablet_bootstrap.cc:492] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: Bootstrap starting.
I20260926 20:15:49.614460 1381 tablet_bootstrap.cc:492] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: Bootstrap starting.
I20260926 20:15:49.614931 1380 tablet_bootstrap.cc:654] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:49.615010 1382 tablet_bootstrap.cc:654] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:49.615382 1381 tablet_bootstrap.cc:654] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:49.616753 1380 tablet_bootstrap.cc:492] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: No bootstrap required, opened a new log
I20260926 20:15:49.616757 1382 tablet_bootstrap.cc:492] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: No bootstrap required, opened a new log
I20260926 20:15:49.616839 1382 ts_tablet_manager.cc:1397] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:49.616839 1380 ts_tablet_manager.cc:1397] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:49.617353 1382 raft_consensus.cc:348] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.617331 1380 raft_consensus.cc:348] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.617465 1382 raft_consensus.cc:374] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:49.617466 1380 raft_consensus.cc:374] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:49.617504 1380 raft_consensus.cc:729] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9ca210023e4f43bb53c1da57b7a5bb, State: Initialized, Role: FOLLOWER
I20260926 20:15:49.617504 1382 raft_consensus.cc:729] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd02987a2167478d83002b643b15cd6b, State: Initialized, Role: FOLLOWER
I20260926 20:15:49.617630 1380 consensus_queue.cc:260] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.617630 1382 consensus_queue.cc:260] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.617906 1382 ts_tablet_manager.cc:1428] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:15:49.617903 1380 ts_tablet_manager.cc:1428] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:15:49.618135 1386 tablet_replica.cc:439] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } }
I20260926 20:15:49.618289 1386 tablet_replica.cc:442] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:49.618446 1381 tablet_bootstrap.cc:492] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: No bootstrap required, opened a new log
I20260926 20:15:49.618539 1381 ts_tablet_manager.cc:1397] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 20:15:49.618520 1388 tablet_replica.cc:439] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } }
I20260926 20:15:49.618676 1388 tablet_replica.cc:442] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:49.619333 1381 raft_consensus.cc:348] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.619474 1381 raft_consensus.cc:374] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:49.619510 1381 raft_consensus.cc:729] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e737227cbe1648abb1c842d79eba7e47, State: Initialized, Role: FOLLOWER
I20260926 20:15:49.619638 1381 consensus_queue.cc:260] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.619928 1381 ts_tablet_manager.cc:1428] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 20:15:49.620186 1387 tablet_replica.cc:439] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } }
I20260926 20:15:49.620314 1387 tablet_replica.cc:442] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:49.637473 1388 raft_consensus.cc:482] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:15:49.637650 1388 raft_consensus.cc:504] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.639007 1388 leader_election.cc:288] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9ca210023e4f43bb53c1da57b7a5bb (127.31.119.193:42195), e737227cbe1648abb1c842d79eba7e47 (127.31.119.195:34669)
I20260926 20:15:49.643728 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.639619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:33393 (local address 127.31.119.195:34669)
0926 20:15:49.639731 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.639737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.639962 (+ 225us) server_negotiation.cc:407] Connection header received
0926 20:15:49.640144 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.640147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.640198 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.640260 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:49.640514 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.641213 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.642415 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.642951 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.643073 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.643128 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.643492 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.643575 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:15:49.644001 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.639554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:36207 (local address 127.31.119.193:42195)
0926 20:15:49.639678 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.639685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.639701 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:15:49.639757 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.639760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.639829 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.639911 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:49.640217 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.641032 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.642675 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.643296 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.643407 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.643473 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.643769 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.643881 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:15:49.644182 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.639244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:34669 (local address 127.31.119.194:33393)
0926 20:15:49.639879 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.639895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.640029 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.640276 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.640279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.640294 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:49.640389 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.640394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.641352 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.641357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.642269 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.642297 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.643212 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.643216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.643333 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.643374 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.643924 (+ 550us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.644014 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":38,"thread_start_us":94,"threads_started":1}
I20260926 20:15:49.644593 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.639181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:42195 (local address 127.31.119.194:36207)
0926 20:15:49.639522 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.639539 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.639649 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.639943 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.639946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.639968 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:49.640107 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.640113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.641154 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.641157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.642547 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.642558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.643527 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.643531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.643616 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.643635 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.644324 (+ 689us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.644403 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":24,"thread_start_us":138,"threads_started":1}
I20260926 20:15:49.644797 1311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4feea81f7d04d0087f0ae49219b4ead" candidate_uuid: "fd02987a2167478d83002b643b15cd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e737227cbe1648abb1c842d79eba7e47" is_pre_election: true
I20260926 20:15:49.644973 1311 raft_consensus.cc:2454] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd02987a2167478d83002b643b15cd6b in term 0.
I20260926 20:15:49.645094 1019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4feea81f7d04d0087f0ae49219b4ead" candidate_uuid: "fd02987a2167478d83002b643b15cd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" is_pre_election: true
I20260926 20:15:49.645246 1019 raft_consensus.cc:2454] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd02987a2167478d83002b643b15cd6b in term 0.
I20260926 20:15:49.645463 1103 leader_election.cc:302] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e737227cbe1648abb1c842d79eba7e47, fd02987a2167478d83002b643b15cd6b; no voters:
I20260926 20:15:49.645776 1388 raft_consensus.cc:2781] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:15:49.645830 1388 raft_consensus.cc:482] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:15:49.645866 1388 raft_consensus.cc:3037] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:49.646919 1388 raft_consensus.cc:504] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.647341 1388 leader_election.cc:288] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [CANDIDATE]: Term 1 election: Requested vote from peers 8a9ca210023e4f43bb53c1da57b7a5bb (127.31.119.193:42195), e737227cbe1648abb1c842d79eba7e47 (127.31.119.195:34669)
I20260926 20:15:49.647683 1019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4feea81f7d04d0087f0ae49219b4ead" candidate_uuid: "fd02987a2167478d83002b643b15cd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb"
I20260926 20:15:49.647753 1311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4feea81f7d04d0087f0ae49219b4ead" candidate_uuid: "fd02987a2167478d83002b643b15cd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e737227cbe1648abb1c842d79eba7e47"
I20260926 20:15:49.647827 1019 raft_consensus.cc:3037] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:49.647890 1311 raft_consensus.cc:3037] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:49.648810 1019 raft_consensus.cc:2454] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd02987a2167478d83002b643b15cd6b in term 1.
I20260926 20:15:49.648844 1311 raft_consensus.cc:2454] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd02987a2167478d83002b643b15cd6b in term 1.
I20260926 20:15:49.649142 1102 leader_election.cc:302] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9ca210023e4f43bb53c1da57b7a5bb, fd02987a2167478d83002b643b15cd6b; no voters:
I20260926 20:15:49.649403 1388 raft_consensus.cc:2781] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:49.649629 1388 raft_consensus.cc:686] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [term 1 LEADER]: Becoming Leader. State: Replica: fd02987a2167478d83002b643b15cd6b, State: Running, Role: LEADER
I20260926 20:15:49.649933 1388 consensus_queue.cc:237] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } }
I20260926 20:15:49.651295 1423 tablet_replica.cc:439] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: TxnStatusTablet state changed. Reason: New leader fd02987a2167478d83002b643b15cd6b. Latest consensus state: current_term: 1 leader_uuid: "fd02987a2167478d83002b643b15cd6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } }
I20260926 20:15:49.651432 1423 tablet_replica.cc:442] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:15:49.651691 1425 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:15:49.651858 1425 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:15:49.652493 874 catalog_manager.cc:5495] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b reported cstate change: term changed from 0 to 1, leader changed from <none> to fd02987a2167478d83002b643b15cd6b (127.31.119.194). New cstate: current_term: 1 leader_uuid: "fd02987a2167478d83002b643b15cd6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } health_report { overall_health: UNKNOWN } } }
I20260926 20:15:49.670702 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.665723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38581 (local address 127.0.0.1:47740)
0926 20:15:49.665912 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.665929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.666041 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.666522 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.666528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.666539 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:15:49.666684 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.666691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.667613 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.667616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.668661 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.668673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.669187 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.669191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.669344 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.669414 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:15:49.670329 (+ 915us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:15:49.670348 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.670424 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.670535 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 20:15:49.670722 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.665847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.31.119.194:38581)
0926 20:15:49.665975 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.665979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.665993 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:15:49.666277 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.666281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.666371 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.666489 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:49.666898 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.667496 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.668825 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.669067 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.669160 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.669682 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:49.670232 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:49.670335 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.670473 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.670530 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:15:49.674055 1311 raft_consensus.cc:1261] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47 [term 1 FOLLOWER]: Refusing update from remote peer fd02987a2167478d83002b643b15cd6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:49.674211 1019 raft_consensus.cc:1261] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 FOLLOWER]: Refusing update from remote peer fd02987a2167478d83002b643b15cd6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:49.675097 1423 consensus_queue.cc:1035] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:49.675623 1423 consensus_queue.cc:1035] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:49.678453 1387 tablet_replica.cc:439] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: TxnStatusTablet state changed. Reason: New leader fd02987a2167478d83002b643b15cd6b. Latest consensus state: current_term: 1 leader_uuid: "fd02987a2167478d83002b643b15cd6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } }
I20260926 20:15:49.678586 1387 tablet_replica.cc:442] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:49.679816 1388 tablet_replica.cc:439] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd02987a2167478d83002b643b15cd6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } }
I20260926 20:15:49.679936 1388 tablet_replica.cc:442] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:15:49.680030 1396 mvcc.cc:205] Tried to move back new op lower bound from 7333698558659297280 to 7333698558569009152. Current Snapshot: MvccSnapshot[applied={T|T < 7333698558659297280 or (T in {7333698558659297280})}]
I20260926 20:15:49.681212 1387 tablet_replica.cc:439] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd02987a2167478d83002b643b15cd6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } }
I20260926 20:15:49.681356 1387 tablet_replica.cc:442] T b4feea81f7d04d0087f0ae49219b4ead P e737227cbe1648abb1c842d79eba7e47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:49.684386 1386 tablet_replica.cc:439] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: TxnStatusTablet state changed. Reason: New leader fd02987a2167478d83002b643b15cd6b. Latest consensus state: current_term: 1 leader_uuid: "fd02987a2167478d83002b643b15cd6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } }
I20260926 20:15:49.684566 1386 tablet_replica.cc:442] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:49.691260 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.686855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.31.119.195:34669)
0926 20:15:49.686958 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.686962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.686974 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:15:49.687053 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.687056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.687111 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.687221 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:49.687535 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.688412 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.689574 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.689922 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.690029 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.690302 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:49.690858 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:49.690971 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.691061 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.691119 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:15:49.694873 1388 tablet_replica.cc:439] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd02987a2167478d83002b643b15cd6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } }
I20260926 20:15:49.695019 1388 tablet_replica.cc:442] T b4feea81f7d04d0087f0ae49219b4ead P fd02987a2167478d83002b643b15cd6b: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:15:49.695937 1386 tablet_replica.cc:439] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd02987a2167478d83002b643b15cd6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 } } peers { permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 } } peers { permanent_uuid: "e737227cbe1648abb1c842d79eba7e47" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34669 } } }
I20260926 20:15:49.696074 1386 tablet_replica.cc:442] T b4feea81f7d04d0087f0ae49219b4ead P 8a9ca210023e4f43bb53c1da57b7a5bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:49.706633 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.702687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:49011 (local address 127.31.119.194:38581)
0926 20:15:49.702791 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.702795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.702809 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:15:49.702858 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.702861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.702907 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.702984 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:49.703277 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.704082 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.705327 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.705691 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.705789 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.706117 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:49.706200 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:49.706270 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.706412 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.706473 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:15:49.706701 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.702310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38581 (local address 127.31.119.195:49011)
0926 20:15:49.702637 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.702653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.702753 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.703003 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.703006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.703013 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:15:49.703136 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.703141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.704194 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.704198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.705166 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.705179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.705806 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.705809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.705913 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.705972 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:15:49.706290 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:15:49.706298 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.706388 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.706503 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
I20260926 20:15:49.713617 1019 raft_consensus.cc:1261] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 FOLLOWER]: Refusing update from remote peer e737227cbe1648abb1c842d79eba7e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:49.713707 1171 raft_consensus.cc:1261] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b [term 1 FOLLOWER]: Refusing update from remote peer e737227cbe1648abb1c842d79eba7e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:49.714131 1387 consensus_queue.cc:1035] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd02987a2167478d83002b643b15cd6b" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:49.714432 1410 consensus_queue.cc:1035] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9ca210023e4f43bb53c1da57b7a5bb" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 42195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:49.738394 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.733965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:54913 (local address 127.31.119.195:34669)
0926 20:15:49.734263 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.734267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.734365 (+ 98us) server_negotiation.cc:407] Connection header received
0926 20:15:49.734563 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.734566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.734620 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.734712 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:49.734986 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.735765 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.736932 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.737350 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.737458 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.737921 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:49.738009 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:49.738052 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.738217 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.738279 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260926 20:15:49.738507 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.733940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:34669 (local address 127.31.119.194:54913)
0926 20:15:49.734285 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:49.734302 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:49.734438 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:49.734723 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:49.734725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:49.734734 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:15:49.734873 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.734880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.735901 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.735904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.736791 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:49.736803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.737579 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.737583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:49.737694 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.737752 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:15:49.738112 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:15:49.738117 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:15:49.738180 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 20:15:49.738264 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":120,"threads_started":1}
I20260926 20:15:49.766436 874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:49.766654 874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:49.769184 874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68a35a9f67124c5998055f499417216b: Sending DeleteTablet for 3 replicas of tablet 9df67288e4a845beb992cb58787aaf34
I20260926 20:15:49.769894 995 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df67288e4a845beb992cb58787aaf34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:49 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:51528
I20260926 20:15:49.770109 1151 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df67288e4a845beb992cb58787aaf34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:49 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:47732
I20260926 20:15:49.770182 1291 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df67288e4a845beb992cb58787aaf34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:49 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:52312
I20260926 20:15:49.770234 1404 tablet_replica.cc:330] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb: stopping tablet replica
I20260926 20:15:49.770382 1404 raft_consensus.cc:2229] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:49.770401 1405 tablet_replica.cc:330] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47: stopping tablet replica
I20260926 20:15:49.770516 1404 raft_consensus.cc:2258] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:49.770519 1405 raft_consensus.cc:2229] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:15:49.770694 1403 tablet_replica.cc:330] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b: stopping tablet replica
I20260926 20:15:49.770807 1403 raft_consensus.cc:2229] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:49.770829 1405 raft_consensus.cc:2258] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:49.770912 1403 raft_consensus.cc:2258] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:49.771823 1404 ts_tablet_manager.cc:1905] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:49.772185 1403 ts_tablet_manager.cc:1905] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:49.774055 1405 ts_tablet_manager.cc:1905] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:49.775061 1404 ts_tablet_manager.cc:1918] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:15:49.775134 1404 log.cc:1196] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/9df67288e4a845beb992cb58787aaf34
I20260926 20:15:49.775385 1404 ts_tablet_manager.cc:1939] T 9df67288e4a845beb992cb58787aaf34 P 8a9ca210023e4f43bb53c1da57b7a5bb: Deleting consensus metadata
I20260926 20:15:49.777951 858 catalog_manager.cc:4841] TS 8a9ca210023e4f43bb53c1da57b7a5bb (127.31.119.193:42195): tablet 9df67288e4a845beb992cb58787aaf34 (table test-table [id=a2599555cb45475099ecca29fcd4d24c]) successfully deleted
I20260926 20:15:49.780306 1403 ts_tablet_manager.cc:1918] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:15:49.780403 1403 log.cc:1196] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/9df67288e4a845beb992cb58787aaf34
I20260926 20:15:49.780665 1403 ts_tablet_manager.cc:1939] T 9df67288e4a845beb992cb58787aaf34 P fd02987a2167478d83002b643b15cd6b: Deleting consensus metadata
I20260926 20:15:49.781497 860 catalog_manager.cc:4841] TS fd02987a2167478d83002b643b15cd6b (127.31.119.194:38581): tablet 9df67288e4a845beb992cb58787aaf34 (table test-table [id=a2599555cb45475099ecca29fcd4d24c]) successfully deleted
I20260926 20:15:49.782159 1405 ts_tablet_manager.cc:1918] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:15:49.782251 1405 log.cc:1196] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/9df67288e4a845beb992cb58787aaf34
I20260926 20:15:49.782508 1405 ts_tablet_manager.cc:1939] T 9df67288e4a845beb992cb58787aaf34 P e737227cbe1648abb1c842d79eba7e47: Deleting consensus metadata
I20260926 20:15:49.783319 859 catalog_manager.cc:4841] TS e737227cbe1648abb1c842d79eba7e47 (127.31.119.195:34669): tablet 9df67288e4a845beb992cb58787aaf34 (table test-table [id=a2599555cb45475099ecca29fcd4d24c]) successfully deleted
Sep 26 20:15:49 dist-test-slave-vnbh krb5kdc[829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453749, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:15:49.796945 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.783523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.31.119.193:42195)
0926 20:15:49.783663 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.783668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.784183 (+ 515us) server_negotiation.cc:407] Connection header received
0926 20:15:49.784332 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.784335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.784390 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.784470 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:49.785646 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.786484 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.787742 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.788091 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.788209 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.792549 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:49.792567 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:49.792585 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:49.792620 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:49.794805 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.795370 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.795376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.795378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.795478 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.795801 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.795804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.795806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.796150 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:49.796359 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.796629 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.796769 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 20:15:49.797710 1058 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544
I20260926 20:15:49.810276 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:49.799718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.31.119.254:40921)
0926 20:15:49.799892 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:49.799896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:49.799978 (+ 82us) server_negotiation.cc:407] Connection header received
0926 20:15:49.800297 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:49.800303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:49.800384 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:49.800449 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:49.801545 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.802336 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.803530 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:49.803863 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:49.803953 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:49.806026 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:49.806030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:49.806033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:49.806067 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:49.807975 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.808563 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.808567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.808569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.808671 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:49.809040 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:49.809044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:49.809046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:49.809347 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:49.809463 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:49.809878 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 20:15:49.810022 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260926 20:15:49.811514 874 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50184
I20260926 20:15:49.813139 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 946
I20260926 20:15:49.821012 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 1086
I20260926 20:15:49.829993 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 1226
I20260926 20:15:49.836211 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 842
2026-09-26T20:15:49Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2893 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:15:49 dist-test-slave-vnbh krb5kdc[1447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:15:49 dist-test-slave-vnbh krb5kdc[1447](info): set up 2 sockets
Sep 26 20:15:49 dist-test-slave-vnbh krb5kdc[1447](info): commencing operation
krb5kdc: starting...
W20260926 20:15:50.887439 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:15:50 dist-test-slave-vnbh krb5kdc[1447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:15:51.028009 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:36659
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:36659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:51.144090 1460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:51.144397 1460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:51.144470 1460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:51.148758 1460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:51.148844 1460 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:15:51.148862 1460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:51.148881 1460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:51.148897 1460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:51.148913 1460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:51.154687 1460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:36659
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:36659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:51.155821 1460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:51.156921 1460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:51.166317 1468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:51.166320 1466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:51.166313 1465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:51.166358 1460 server_base.cc:1034] running on GCE node
I20260926 20:15:51.168471 1460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:51.169767 1460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:51.170962 1460 hybrid_clock.cc:648] HybridClock initialized: now 1790453751170923 us; error 57 us; skew 500 ppm
Sep 26 20:15:51 dist-test-slave-vnbh krb5kdc[1447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453751, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:51.173995 1460 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:15:51.175000 1460 webserver.cc:460] Webserver started at http://127.31.119.254:36631/ using document root <none> and password file <none>
I20260926 20:15:51.175603 1460 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:51.175698 1460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:51.175930 1460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:51.178102 1460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "224e23cad6554d83beed9171478b2144"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "68691aba1605be6c0e715b4ed154a306"
server_key_iv: "deb3047abf347d52ed44f65f0dc9fac1"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.178690 1460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "224e23cad6554d83beed9171478b2144"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "68691aba1605be6c0e715b4ed154a306"
server_key_iv: "deb3047abf347d52ed44f65f0dc9fac1"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.182682 1460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:15:51.185386 1475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:51.186919 1460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:15:51.187098 1460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "224e23cad6554d83beed9171478b2144"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "68691aba1605be6c0e715b4ed154a306"
server_key_iv: "deb3047abf347d52ed44f65f0dc9fac1"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.187233 1460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:51.214039 1460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:51.217631 1460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:51.217886 1460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:51.226666 1460 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:36659
I20260926 20:15:51.226683 1537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:36659 every 8 connection(s)
I20260926 20:15:51.227777 1460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:15:51.230942 1538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:51.235442 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 1460
I20260926 20:15:51.235563 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:15:51.235843 32223 external_mini_cluster.cc:1428] Setting key 424330903c2f9446245b7164fb7e892c
I20260926 20:15:51.237388 1538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144: Bootstrap starting.
I20260926 20:15:51.239848 1538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:51.240640 1538 log.cc:824] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:51.242776 1538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144: No bootstrap required, opened a new log
Sep 26 20:15:51 dist-test-slave-vnbh krb5kdc[1447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:51.246941 1538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224e23cad6554d83beed9171478b2144" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36659 } }
I20260926 20:15:51.247264 1538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:51.247342 1538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224e23cad6554d83beed9171478b2144, State: Initialized, Role: FOLLOWER
I20260926 20:15:51.248081 1538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224e23cad6554d83beed9171478b2144" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36659 } }
I20260926 20:15:51.248239 1538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:15:51.248344 1538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:15:51.248481 1538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:51.249571 1538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224e23cad6554d83beed9171478b2144" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36659 } }
I20260926 20:15:51.250097 1538 leader_election.cc:302] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 224e23cad6554d83beed9171478b2144; no voters:
I20260926 20:15:51.250613 1538 leader_election.cc:288] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:15:51.250703 1543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:51.251039 1543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [term 1 LEADER]: Becoming Leader. State: Replica: 224e23cad6554d83beed9171478b2144, State: Running, Role: LEADER
I20260926 20:15:51.251466 1543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224e23cad6554d83beed9171478b2144" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36659 } }
I20260926 20:15:51.252385 1538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:51.252704 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:51.237552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.31.119.254:36659)
0926 20:15:51.238080 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:51.238098 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:51.238135 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:15:51.238837 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:51.238856 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:51.239308 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:51.239670 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:51.240517 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.241696 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.242577 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.243128 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.243319 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:51.246195 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:51.246201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:51.246218 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:51.246250 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:51.248757 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:51.249274 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:51.249328 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:51.249337 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:51.249480 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:51.249814 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:51.249818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:51.249820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:51.250444 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:51.250767 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:51.251235 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 20:15:51.251593 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":141,"threads_started":1}
I20260926 20:15:51.253378 1544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 224e23cad6554d83beed9171478b2144. Latest consensus state: current_term: 1 leader_uuid: "224e23cad6554d83beed9171478b2144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224e23cad6554d83beed9171478b2144" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36659 } } }
I20260926 20:15:51.253582 1544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:51.252658 1543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "224e23cad6554d83beed9171478b2144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224e23cad6554d83beed9171478b2144" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36659 } } }
I20260926 20:15:51.254012 1543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:51.254185 1552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:15:51.257498 1552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:15:51.264308 1552 catalog_manager.cc:1347] Generated new cluster ID: d83a54d37aa24bd59ec07a677dfef618
I20260926 20:15:51.264420 1552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:15:51.272356 1552 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:15:51.273597 1552 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:15:51.278975 1552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144: Generated new TSK 0
I20260926 20:15:51.279928 1552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:15:51.313265 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:36659
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:51.448203 1564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:51.448518 1564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:51.448568 1564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:51.452853 1564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:51.452922 1564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:51.452965 1564 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:15:51.452979 1564 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:15:51.453013 1564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:15:51.458730 1564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.119.254:36659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:51.459831 1564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:51.460929 1564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:51.470866 1572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:51.470866 1570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:51.471020 1569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:51.471095 1564 server_base.cc:1034] running on GCE node
I20260926 20:15:51.471624 1564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:51.472297 1564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:51.473482 1564 hybrid_clock.cc:648] HybridClock initialized: now 1790453751473465 us; error 32 us; skew 500 ppm
Sep 26 20:15:51 dist-test-slave-vnbh krb5kdc[1447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453751, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:51.476555 1564 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:15:51.477586 1564 webserver.cc:460] Webserver started at http://127.31.119.193:40615/ using document root <none> and password file <none>
I20260926 20:15:51.478178 1564 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:51.478271 1564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:51.478497 1564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:51.480580 1564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "6a1173b942ea4d4897ea419bac852baa"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "a09bf813c4ede225163fa3a571101523"
server_key_iv: "9134501ab2abc25b9a2afb754a96602b"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.481175 1564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "6a1173b942ea4d4897ea419bac852baa"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "a09bf813c4ede225163fa3a571101523"
server_key_iv: "9134501ab2abc25b9a2afb754a96602b"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.484917 1564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:15:51.487555 1579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:51.488864 1564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:15:51.488961 1564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "6a1173b942ea4d4897ea419bac852baa"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "a09bf813c4ede225163fa3a571101523"
server_key_iv: "9134501ab2abc25b9a2afb754a96602b"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.489063 1564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:51.514724 1564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:51.518311 1564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:51.518479 1564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:51.520002 1564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:51.520093 1564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:51.520170 1564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:51.520213 1564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:51.530864 1564 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:40245
I20260926 20:15:51.530903 1694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:40245 every 8 connection(s)
Sep 26 20:15:51 dist-test-slave-vnbh krb5kdc[1447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453751, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:51.531998 1564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:15:51.535436 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:51.521066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:54529 (local address 127.31.119.254:36659)
0926 20:15:51.521218 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:51.521223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:51.522479 (+ 1256us) server_negotiation.cc:407] Connection header received
0926 20:15:51.523809 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:51.523814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:51.523886 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:51.523997 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:51.525750 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.526667 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.528027 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.528432 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.528559 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:51.531643 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:51.531646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:51.531649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:51.531731 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:51.533393 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:51.533912 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:51.533915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:51.533917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:51.534001 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:51.534339 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:51.534342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:51.534343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:51.534533 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:51.534638 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:51.535126 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 20:15:51.535261 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:15:51.536218 1588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:51.521760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36659 (local address 127.31.119.193:54529)
0926 20:15:51.522315 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:51.522361 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:51.523516 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:51.524193 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:51.524201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:51.524756 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:51.525575 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:51.525593 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.526840 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.526843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:51.527905 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:51.527915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.528634 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.528639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:51.528909 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:51.529631 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:51.529660 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:51.531476 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:51.533506 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:51.533512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:51.533527 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:51.533797 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:51.534090 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:51.534094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:51.534096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:51.534238 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:51.534634 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:51.534641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:51.534894 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 20:15:51.535073 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 20:15:51.535398 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":143,"threads_started":1}
I20260926 20:15:51.537762 1695 heartbeater.cc:347] Connected to a master server at 127.31.119.254:36659
I20260926 20:15:51.537938 1695 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:51.538502 1695 heartbeater.cc:511] Master 127.31.119.254:36659 requested a full tablet report, sending...
I20260926 20:15:51.540215 1488 ts_manager.cc:194] Registered new tserver with Master: 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245)
I20260926 20:15:51.541652 1488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:54529
I20260926 20:15:51.541837 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 1564
I20260926 20:15:51.541960 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:15:51.542234 32223 external_mini_cluster.cc:1428] Setting key 8ab1d239eec7c80f3c15898f5b3a3f09
I20260926 20:15:51.548280 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:51.539741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36659 (local address 127.31.119.193:49103)
0926 20:15:51.540023 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:51.540039 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:51.540148 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:51.540377 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:51.540380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:51.540598 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:51.540719 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:51.540726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.541713 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.541717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:51.542518 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:51.542528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.543119 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.543123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:51.543214 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:51.543806 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:51.543821 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:51.544257 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:51.546657 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:51.546661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:51.546663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:51.546892 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:51.547217 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:51.547219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:51.547220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:51.547308 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:51.547779 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:51.547781 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:51.547865 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 20:15:51.547954 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:15:51.548077 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":3,"thread_start_us":110,"threads_started":1}
I20260926 20:15:51.548437 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:51.539806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:49103 (local address 127.31.119.254:36659)
0926 20:15:51.539939 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:51.539944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:51.540102 (+ 158us) server_negotiation.cc:407] Connection header received
0926 20:15:51.540230 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:51.540232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:51.540268 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:51.540359 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:51.540855 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.541576 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.542646 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.542988 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.543084 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:51.544412 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:51.544416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:51.544418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:51.544442 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:51.546514 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:51.546999 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:51.547006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:51.547011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:51.547106 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:51.547399 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:51.547405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:51.547409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:51.547670 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:51.547775 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:51.548036 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 20:15:51.548184 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:15:51.572769 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:36659
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:51.688267 1704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:51.688567 1704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:51.688651 1704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:51.693034 1704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:51.693125 1704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:51.693215 1704 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:15:51.693254 1704 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:15:51.693332 1704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:15:51.699297 1704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.119.254:36659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:51.700376 1704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:51.701462 1704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:51.711882 1709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:51.711867 1710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:51.711867 1712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:51.711963 1704 server_base.cc:1034] running on GCE node
I20260926 20:15:51.712646 1704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:51.713475 1704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:51.714655 1704 hybrid_clock.cc:648] HybridClock initialized: now 1790453751714628 us; error 42 us; skew 500 ppm
Sep 26 20:15:51 dist-test-slave-vnbh krb5kdc[1447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453751, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:51.717971 1704 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:15:51.719013 1704 webserver.cc:460] Webserver started at http://127.31.119.194:35009/ using document root <none> and password file <none>
I20260926 20:15:51.719684 1704 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:51.719789 1704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:51.720036 1704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:51.722407 1704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "7ec718ec869445058edcf766f03b5121"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "a87b9880871a880c9ca4ec93f4b05eb6"
server_key_iv: "775162e7674ddb8a63c9165019ec7cf4"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.723081 1704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "7ec718ec869445058edcf766f03b5121"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "a87b9880871a880c9ca4ec93f4b05eb6"
server_key_iv: "775162e7674ddb8a63c9165019ec7cf4"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.727421 1704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:15:51.730187 1719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:51.731515 1704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:15:51.731658 1704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "7ec718ec869445058edcf766f03b5121"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "a87b9880871a880c9ca4ec93f4b05eb6"
server_key_iv: "775162e7674ddb8a63c9165019ec7cf4"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.731779 1704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:51.766729 1704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:51.770397 1704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:51.770609 1704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:51.772125 1704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:51.772197 1704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:51.772264 1704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:51.772308 1704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:15:51 dist-test-slave-vnbh krb5kdc[1447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453751, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:51.783761 1704 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:37329
I20260926 20:15:51.783771 1834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:37329 every 8 connection(s)
I20260926 20:15:51.784968 1704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:15:51.787873 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:51.773421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:50343 (local address 127.31.119.254:36659)
0926 20:15:51.773557 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:51.773561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:51.774777 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 20:15:51.776068 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:51.776072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:51.776135 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:51.776251 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:51.778100 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.778837 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.780273 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.780603 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.780723 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:51.783629 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:51.783633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:51.783636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:51.783669 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:51.785254 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:51.786027 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:51.786031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:51.786033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:51.786142 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:51.786538 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:51.786541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:51.786543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:51.786798 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:51.786898 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:51.787562 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 20:15:51.787750 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:15:51.788646 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:51.773968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36659 (local address 127.31.119.194:50343)
0926 20:15:51.774510 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:51.774666 (+ 156us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:51.775828 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:51.776456 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:51.776468 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:51.777025 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:51.777854 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:51.777878 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.778962 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.778965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:51.780131 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:51.780141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.780803 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.780810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:51.781049 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:51.781833 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:51.781869 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:51.783442 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:51.785400 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:51.785406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:51.785420 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:51.785919 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:51.786235 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:51.786238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:51.786241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:51.786406 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:51.786948 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:51.786953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:51.787207 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 20:15:51.787375 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 20:15:51.787709 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":147,"threads_started":1}
I20260926 20:15:51.790025 1835 heartbeater.cc:347] Connected to a master server at 127.31.119.254:36659
I20260926 20:15:51.790202 1835 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:51.790699 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 1704
I20260926 20:15:51.790799 1835 heartbeater.cc:511] Master 127.31.119.254:36659 requested a full tablet report, sending...
I20260926 20:15:51.790848 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:15:51.791229 32223 external_mini_cluster.cc:1428] Setting key 8251b2aaad30a226b68ec6b9de9a749c
I20260926 20:15:51.792047 1492 ts_manager.cc:194] Registered new tserver with Master: 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329)
I20260926 20:15:51.792807 1492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:50343
I20260926 20:15:51.800968 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:51.792605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:58991 (local address 127.31.119.254:36659)
0926 20:15:51.793128 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:51.793134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:51.793150 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:15:51.793191 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:51.793197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:51.793245 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:51.793358 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:51.794070 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.794811 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.795647 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.795898 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.795981 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:51.797428 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:51.797431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:51.797434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:51.797455 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:51.798950 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:51.799487 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:51.799490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:51.799491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:51.799572 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:51.799948 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:51.799951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:51.799952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:51.800232 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:51.800323 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:51.800691 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 20:15:51.800800 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451}
I20260926 20:15:51.801040 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:51.792561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36659 (local address 127.31.119.194:58991)
0926 20:15:51.792832 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:51.792845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:51.792948 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:51.793493 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:51.793500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:51.793756 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:51.793913 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:51.793924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.794943 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.794945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:51.795500 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:51.795506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:51.796021 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:51.796034 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:51.796164 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:51.796722 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:51.796742 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:51.797268 (+ 526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:51.799074 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:51.799081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:51.799085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:51.799367 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:51.799679 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:51.799682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:51.799686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:51.799804 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:51.800345 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:51.800349 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:51.800474 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 20:15:51.800599 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 20:15:51.800788 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:15:51.821434 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:36659
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:51.938473 1844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:51.938764 1844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:51.938813 1844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:51.942823 1844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:51.942886 1844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:51.942927 1844 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:15:51.942941 1844 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:15:51.942973 1844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:15:51.948427 1844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.119.254:36659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:51.949679 1844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:51.950744 1844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:15:51.961374 1844 server_base.cc:1034] running on GCE node
W20260926 20:15:51.961318 1849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:51.961323 1852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:51.961333 1850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:51.962157 1844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:51.962854 1844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:51.964041 1844 hybrid_clock.cc:648] HybridClock initialized: now 1790453751964025 us; error 29 us; skew 500 ppm
Sep 26 20:15:51 dist-test-slave-vnbh krb5kdc[1447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453751, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:51.967271 1844 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:15:51.968318 1844 webserver.cc:460] Webserver started at http://127.31.119.195:36537/ using document root <none> and password file <none>
I20260926 20:15:51.968914 1844 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:51.968976 1844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:51.969159 1844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:51.971495 1844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "0ac79daf846b410091bc714297b793ee"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "f21b22300d9bfaf4f25926fe2080629a"
server_key_iv: "e0961ee5a28a090be3c3671cc7c382aa"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.972072 1844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "0ac79daf846b410091bc714297b793ee"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "f21b22300d9bfaf4f25926fe2080629a"
server_key_iv: "e0961ee5a28a090be3c3671cc7c382aa"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.976297 1844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:15:51.978943 1859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:51.980285 1844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 20:15:51.980387 1844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "0ac79daf846b410091bc714297b793ee"
format_stamp: "Formatted at 2026-09-26 20:15:51 on dist-test-slave-vnbh"
server_key: "f21b22300d9bfaf4f25926fe2080629a"
server_key_iv: "e0961ee5a28a090be3c3671cc7c382aa"
server_key_version: "encryptionkey@0"
I20260926 20:15:51.980540 1844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:51.998180 1844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:52.001924 1844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:52.002146 1844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:52.003729 1844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:52.003784 1844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:52.003829 1844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:52.003880 1844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:15:52 dist-test-slave-vnbh krb5kdc[1447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453751, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:52.015409 1844 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:36951
I20260926 20:15:52.015492 1974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:36951 every 8 connection(s)
I20260926 20:15:52.016695 1844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:15:52.018535 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 1844
I20260926 20:15:52.018680 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:15:52.019032 32223 external_mini_cluster.cc:1428] Setting key d831081a27b1d0ded8730cd40aaa48b0
I20260926 20:15:52.019850 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.004829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:60071 (local address 127.31.119.254:36659)
0926 20:15:52.004986 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.004991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.006158 (+ 1167us) server_negotiation.cc:407] Connection header received
0926 20:15:52.007684 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.007688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.007765 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.007876 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:52.009889 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.010725 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.012163 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.012410 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.012524 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.015257 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:52.015262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:52.015266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:52.015308 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:52.017175 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:52.017864 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:52.017866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:52.017868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:52.017943 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:52.018340 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:52.018345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:52.018347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:52.018620 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:52.018922 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.019560 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.019678 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:15:52.020823 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.005368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36659 (local address 127.31.119.195:60071)
0926 20:15:52.005941 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.006010 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.007369 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.008137 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.008147 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.008763 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:52.009633 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.009658 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.010874 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.010877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.012013 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.012024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.012573 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.012577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.012776 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.013467 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:52.013485 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:52.015043 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:52.017378 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:52.017384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:52.017401 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:52.017747 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:52.018027 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:52.018030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:52.018031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:52.018191 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:52.018958 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:52.018964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:52.019301 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.019471 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.019862 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":136,"threads_started":1}
I20260926 20:15:52.021955 1975 heartbeater.cc:347] Connected to a master server at 127.31.119.254:36659
I20260926 20:15:52.022114 1975 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:52.022684 1975 heartbeater.cc:511] Master 127.31.119.254:36659 requested a full tablet report, sending...
I20260926 20:15:52.023737 1492 ts_manager.cc:194] Registered new tserver with Master: 0ac79daf846b410091bc714297b793ee (127.31.119.195:36951)
I20260926 20:15:52.024461 1492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:60071
I20260926 20:15:52.031005 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.024267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36659 (local address 127.31.119.195:46327)
0926 20:15:52.024516 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.024532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.024622 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.025027 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.025030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.025235 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:52.025387 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.025393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.026355 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.026358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.026910 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.026917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.027311 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.027314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.027379 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.027910 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:52.027921 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:52.028245 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:52.029827 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:52.029829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:52.029831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:52.030019 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:52.030246 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:52.030248 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:52.030249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:52.030320 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:52.030609 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:52.030611 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:52.030680 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.030764 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.030869 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
I20260926 20:15:52.031057 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.024736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:46327 (local address 127.31.119.254:36659)
0926 20:15:52.024849 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.024852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.024863 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:15:52.024893 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.024894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.024926 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.024991 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:52.025495 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.026238 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.027008 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.027216 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.027304 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.028359 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:52.028362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:52.028364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:52.028381 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:52.029684 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:52.030104 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:52.030106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:52.030107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:52.030175 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:52.030389 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:52.030390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:52.030392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:52.030540 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:52.030611 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.030862 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.030950 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:15:52.033147 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:15:52 dist-test-slave-vnbh krb5kdc[1447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:15:52 dist-test-slave-vnbh krb5kdc[1447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:52.060079 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.051511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.31.119.254:36659)
0926 20:15:52.051711 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.051715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.051728 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:15:52.051777 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.051779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.051820 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.051909 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:52.052659 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.053276 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.054319 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.054531 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.054609 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.057122 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:52.057125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:52.057127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:52.057153 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:52.058590 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:52.059032 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:52.059035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:52.059036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:52.059109 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:52.059348 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:52.059350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:52.059351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:52.059522 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:52.059621 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.059856 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.059954 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 20:15:52.063649 1488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:15:52.066845 1488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:15:52.082248 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.075697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:37329 (local address 127.0.0.1:54424)
0926 20:15:52.076420 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.076473 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.076607 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.077513 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.077515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.077536 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:52.077780 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.077789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.079254 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.079259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.080368 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.080378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.081506 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.081510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.081674 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.081723 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.081920 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.082071 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":9,"spinlock_wait_cycles":9472,"thread_start_us":79,"threads_started":1}
I20260926 20:15:52.082247 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.076514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.31.119.194:37329)
0926 20:15:52.076971 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.076986 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.077009 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:15:52.077087 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.077093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.077349 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.077927 (+ 578us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:52.078051 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.079148 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.080484 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.081364 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.081526 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.081706 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.082029 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.082121 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":143,"threads_started":1}
I20260926 20:15:52.082993 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.075721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40245 (local address 127.0.0.1:38592)
0926 20:15:52.076653 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.076670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.076760 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.077491 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.077500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.077524 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:52.077702 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.077710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.079163 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.079166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.080442 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.080453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.082484 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.082487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.082691 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.082715 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.082780 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.082851 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":206,"spinlock_wait_cycles":9984,"thread_start_us":121,"threads_started":1}
I20260926 20:15:52.083166 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.075914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.31.119.193:40245)
0926 20:15:52.076896 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.076906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.076931 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:15:52.077010 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.077016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.077254 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.077482 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:52.077844 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.079051 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.081467 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.082354 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.082728 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.082880 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.082930 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.083036 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":100,"threads_started":1}
I20260926 20:15:52.083552 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.076027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:36951 (local address 127.0.0.1:50024)
0926 20:15:52.077121 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.077133 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.077304 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.077771 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.077774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.077792 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:52.077891 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.077899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.078925 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.078929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.080445 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.080453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.082453 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.082460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.083264 (+ 804us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.083291 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.083359 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.083431 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":62,"thread_start_us":91,"threads_started":1}
I20260926 20:15:52.084012 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.076499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.31.119.195:36951)
0926 20:15:52.077001 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.077011 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.077217 (+ 206us) server_negotiation.cc:407] Connection header received
0926 20:15:52.077448 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.077453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.077631 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.077791 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:52.078006 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.078816 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.080546 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.081377 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.081543 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.081709 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.083725 (+ 2016us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.083859 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":102,"threads_started":1}
I20260926 20:15:52.085925 1769 tablet_service.cc:1467] Processing CreateTablet for tablet 1290352da8194162b556692df971fe09 (DEFAULT_TABLE table=test-table [id=e6ca835b957842279d558c77d2368519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:52.086352 1909 tablet_service.cc:1467] Processing CreateTablet for tablet 1290352da8194162b556692df971fe09 (DEFAULT_TABLE table=test-table [id=e6ca835b957842279d558c77d2368519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:52.086956 1769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1290352da8194162b556692df971fe09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:52.087309 1909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1290352da8194162b556692df971fe09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:52.088721 1629 tablet_service.cc:1467] Processing CreateTablet for tablet 1290352da8194162b556692df971fe09 (DEFAULT_TABLE table=test-table [id=e6ca835b957842279d558c77d2368519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:52.089712 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1290352da8194162b556692df971fe09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:52.094327 1998 tablet_bootstrap.cc:492] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: Bootstrap starting.
I20260926 20:15:52.095820 1999 tablet_bootstrap.cc:492] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: Bootstrap starting.
I20260926 20:15:52.096762 2000 tablet_bootstrap.cc:492] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: Bootstrap starting.
I20260926 20:15:52.097059 1998 tablet_bootstrap.cc:654] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:52.098006 1999 tablet_bootstrap.cc:654] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:52.098529 1998 log.cc:824] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:52.098806 1999 log.cc:824] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:52.099094 2000 tablet_bootstrap.cc:654] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:52.099783 2000 log.cc:824] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:52.100749 1998 tablet_bootstrap.cc:492] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: No bootstrap required, opened a new log
I20260926 20:15:52.100968 1998 ts_tablet_manager.cc:1397] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 20:15:52.101009 1999 tablet_bootstrap.cc:492] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: No bootstrap required, opened a new log
I20260926 20:15:52.101193 1999 ts_tablet_manager.cc:1397] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:15:52.101794 2000 tablet_bootstrap.cc:492] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: No bootstrap required, opened a new log
I20260926 20:15:52.101967 2000 ts_tablet_manager.cc:1397] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:15:52.104229 1998 raft_consensus.cc:348] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.104483 1998 raft_consensus.cc:374] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:52.104537 1998 raft_consensus.cc:729] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec718ec869445058edcf766f03b5121, State: Initialized, Role: FOLLOWER
I20260926 20:15:52.105044 1998 consensus_queue.cc:260] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.105572 1999 raft_consensus.cc:348] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.105846 1999 raft_consensus.cc:374] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:52.105903 1999 raft_consensus.cc:729] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac79daf846b410091bc714297b793ee, State: Initialized, Role: FOLLOWER
I20260926 20:15:52.106172 1835 heartbeater.cc:503] Master 127.31.119.254:36659 was elected leader, sending a full tablet report...
I20260926 20:15:52.106328 1999 consensus_queue.cc:260] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.106403 2000 raft_consensus.cc:348] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.106645 2000 raft_consensus.cc:374] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:52.106704 2000 raft_consensus.cc:729] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1173b942ea4d4897ea419bac852baa, State: Initialized, Role: FOLLOWER
I20260926 20:15:52.106155 1998 ts_tablet_manager.cc:1428] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 20:15:52.107152 2000 consensus_queue.cc:260] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.107420 1999 ts_tablet_manager.cc:1428] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:15:52.107748 1975 heartbeater.cc:503] Master 127.31.119.254:36659 was elected leader, sending a full tablet report...
I20260926 20:15:52.107777 1695 heartbeater.cc:503] Master 127.31.119.254:36659 was elected leader, sending a full tablet report...
I20260926 20:15:52.108090 2000 ts_tablet_manager.cc:1428] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 20:15:52.267966 1976 tablet.cc:2369] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:15:52.283963 1696 tablet.cc:2369] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:15:52.287194 1836 tablet.cc:2369] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:15:52.393337 2005 raft_consensus.cc:482] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:15:52.393565 2005 raft_consensus.cc:504] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.394829 2005 leader_election.cc:288] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329), 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245)
I20260926 20:15:52.398932 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.395253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:36115 (local address 127.31.119.193:40245)
0926 20:15:52.395364 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.395370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.395479 (+ 109us) server_negotiation.cc:407] Connection header received
0926 20:15:52.395643 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.395700 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.395750 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.395868 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:52.396236 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.396766 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.397925 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.398360 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.398490 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.398544 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.398722 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.398794 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:15:52.398932 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.395150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40245 (local address 127.31.119.195:36115)
0926 20:15:52.395364 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.395388 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.395502 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.395882 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.395885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.395913 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:52.396109 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.396116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.396889 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.396893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.397793 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.397803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.398485 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.398489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.398596 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.398615 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.398709 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.398790 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":41}
I20260926 20:15:52.399077 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.395193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:54827 (local address 127.31.119.194:37329)
0926 20:15:52.395350 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.395356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.395749 (+ 393us) server_negotiation.cc:407] Connection header received
0926 20:15:52.395817 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.395819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.395857 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.395937 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:52.396599 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.397104 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.398104 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.398509 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.398655 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.398715 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.398877 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.398950 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:15:52.399133 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.395152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:37329 (local address 127.31.119.195:54827)
0926 20:15:52.395574 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.395589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.395679 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.395948 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.395951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.395967 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:52.396471 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.396478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.397216 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.397222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.397977 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.397985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.398628 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.398631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.398747 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.398773 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.398862 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.398933 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":1,"thread_start_us":140,"threads_started":1}
I20260926 20:15:52.399715 1649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1290352da8194162b556692df971fe09" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1173b942ea4d4897ea419bac852baa" is_pre_election: true
I20260926 20:15:52.399775 1787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1290352da8194162b556692df971fe09" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec718ec869445058edcf766f03b5121" is_pre_election: true
I20260926 20:15:52.400053 1649 raft_consensus.cc:2454] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 0.
I20260926 20:15:52.400075 1787 raft_consensus.cc:2454] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 0.
I20260926 20:15:52.400665 1860 leader_election.cc:302] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ac79daf846b410091bc714297b793ee, 6a1173b942ea4d4897ea419bac852baa, 7ec718ec869445058edcf766f03b5121; no voters:
I20260926 20:15:52.400972 2005 raft_consensus.cc:2781] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:15:52.401054 2005 raft_consensus.cc:482] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:15:52.401085 2005 raft_consensus.cc:3037] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:52.402194 2005 raft_consensus.cc:504] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.402606 2005 leader_election.cc:288] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 election: Requested vote from peers 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329), 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245)
I20260926 20:15:52.403007 1787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1290352da8194162b556692df971fe09" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec718ec869445058edcf766f03b5121"
I20260926 20:15:52.403041 1649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1290352da8194162b556692df971fe09" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1173b942ea4d4897ea419bac852baa"
I20260926 20:15:52.403141 1787 raft_consensus.cc:3037] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:52.403234 1649 raft_consensus.cc:3037] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:52.404315 1787 raft_consensus.cc:2454] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 1.
I20260926 20:15:52.404547 1649 raft_consensus.cc:2454] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 1.
I20260926 20:15:52.404742 1862 leader_election.cc:302] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac79daf846b410091bc714297b793ee, 7ec718ec869445058edcf766f03b5121; no voters:
I20260926 20:15:52.404920 2005 raft_consensus.cc:2781] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:52.405161 2005 raft_consensus.cc:686] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 1 LEADER]: Becoming Leader. State: Replica: 0ac79daf846b410091bc714297b793ee, State: Running, Role: LEADER
I20260926 20:15:52.405577 2005 consensus_queue.cc:237] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.408953 1491 catalog_manager.cc:5495] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac79daf846b410091bc714297b793ee (127.31.119.195). New cstate: current_term: 1 leader_uuid: "0ac79daf846b410091bc714297b793ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } health_report { overall_health: HEALTHY } } }
Sep 26 20:15:52 dist-test-slave-vnbh krb5kdc[1447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453751, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:52.472548 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.463330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36659 (local address 127.0.0.1:39606)
0926 20:15:52.463735 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.463748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.463891 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.464162 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.464165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.464602 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:52.464689 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.464696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.465474 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.465480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.466072 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.466081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.466783 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.466787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.466871 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.467464 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:52.467476 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:52.469311 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:52.470905 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:52.470918 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:52.470925 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:52.471171 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:52.471484 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:52.471487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:52.471488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:52.471582 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:52.472020 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:52.472026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:52.472163 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.472262 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.472405 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":126,"threads_started":1}
I20260926 20:15:52.472719 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.463375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39606 (local address 127.31.119.254:36659)
0926 20:15:52.463566 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.463571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.463832 (+ 261us) server_negotiation.cc:407] Connection header received
0926 20:15:52.463987 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.463990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.464032 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.464146 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:52.464806 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.465379 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.466368 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.466601 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.466774 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.469456 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:52.469459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:52.469462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:52.469489 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:52.470799 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:52.471272 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:52.471277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:52.471279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:52.471364 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:52.471713 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:52.471716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:52.471717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:52.471915 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:52.472003 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.472334 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.472483 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":67}
I20260926 20:15:52.477037 1491 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:39606:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:15:52.477902 1491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:15:52.482921 1629 tablet_service.cc:1467] Processing CreateTablet for tablet 2612b6361ef24553a24ad71078591d4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72e4ed2c3e50409a9f8e3d6f35c92c2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:15:52.482920 1769 tablet_service.cc:1467] Processing CreateTablet for tablet 2612b6361ef24553a24ad71078591d4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72e4ed2c3e50409a9f8e3d6f35c92c2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:15:52.482952 1909 tablet_service.cc:1467] Processing CreateTablet for tablet 2612b6361ef24553a24ad71078591d4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72e4ed2c3e50409a9f8e3d6f35c92c2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:15:52.483714 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2612b6361ef24553a24ad71078591d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:52.483714 1769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2612b6361ef24553a24ad71078591d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:52.483795 1909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2612b6361ef24553a24ad71078591d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:52.486698 1999 tablet_bootstrap.cc:492] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: Bootstrap starting.
I20260926 20:15:52.486701 2000 tablet_bootstrap.cc:492] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: Bootstrap starting.
I20260926 20:15:52.486979 1998 tablet_bootstrap.cc:492] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: Bootstrap starting.
I20260926 20:15:52.487759 2000 tablet_bootstrap.cc:654] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:52.487797 1999 tablet_bootstrap.cc:654] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:52.487949 1998 tablet_bootstrap.cc:654] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:52.489184 2000 tablet_bootstrap.cc:492] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: No bootstrap required, opened a new log
I20260926 20:15:52.489210 1998 tablet_bootstrap.cc:492] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: No bootstrap required, opened a new log
I20260926 20:15:52.489303 2000 ts_tablet_manager.cc:1397] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:52.489327 1998 ts_tablet_manager.cc:1397] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:15:52.489396 1999 tablet_bootstrap.cc:492] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: No bootstrap required, opened a new log
I20260926 20:15:52.489482 1999 ts_tablet_manager.cc:1397] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:52.489785 1998 raft_consensus.cc:348] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.489789 2000 raft_consensus.cc:348] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.489933 2000 raft_consensus.cc:374] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:52.489929 1998 raft_consensus.cc:374] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:52.489899 1999 raft_consensus.cc:348] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.489974 1998 raft_consensus.cc:729] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec718ec869445058edcf766f03b5121, State: Initialized, Role: FOLLOWER
I20260926 20:15:52.489974 2000 raft_consensus.cc:729] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1173b942ea4d4897ea419bac852baa, State: Initialized, Role: FOLLOWER
I20260926 20:15:52.490012 1999 raft_consensus.cc:374] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:52.490049 1999 raft_consensus.cc:729] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac79daf846b410091bc714297b793ee, State: Initialized, Role: FOLLOWER
I20260926 20:15:52.490098 1998 consensus_queue.cc:260] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.490097 2000 consensus_queue.cc:260] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.490176 1999 consensus_queue.cc:260] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.490360 2000 ts_tablet_manager.cc:1428] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:15:52.490363 1998 ts_tablet_manager.cc:1428] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:15:52.490475 1999 ts_tablet_manager.cc:1428] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:15:52.490633 2006 tablet_replica.cc:439] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } } }
I20260926 20:15:52.490682 2005 tablet_replica.cc:439] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } } }
I20260926 20:15:52.490785 2006 tablet_replica.cc:442] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:52.490808 2005 tablet_replica.cc:442] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:52.491061 2004 tablet_replica.cc:439] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } } }
I20260926 20:15:52.491195 2004 tablet_replica.cc:442] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:52.586308 2005 raft_consensus.cc:482] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:15:52.586436 2005 raft_consensus.cc:504] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.586932 2005 leader_election.cc:288] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245), 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329)
I20260926 20:15:52.587350 1649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2612b6361ef24553a24ad71078591d4d" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1173b942ea4d4897ea419bac852baa" is_pre_election: true
I20260926 20:15:52.587467 1787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2612b6361ef24553a24ad71078591d4d" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec718ec869445058edcf766f03b5121" is_pre_election: true
I20260926 20:15:52.587517 1649 raft_consensus.cc:2454] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 0.
I20260926 20:15:52.587616 1787 raft_consensus.cc:2454] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 0.
I20260926 20:15:52.587812 1860 leader_election.cc:302] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac79daf846b410091bc714297b793ee, 6a1173b942ea4d4897ea419bac852baa; no voters:
I20260926 20:15:52.587989 2005 raft_consensus.cc:2781] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:15:52.588048 2005 raft_consensus.cc:482] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:15:52.588076 2005 raft_consensus.cc:3037] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:52.589103 2005 raft_consensus.cc:504] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.589497 2005 leader_election.cc:288] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 election: Requested vote from peers 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245), 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329)
I20260926 20:15:52.589895 1649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2612b6361ef24553a24ad71078591d4d" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1173b942ea4d4897ea419bac852baa"
I20260926 20:15:52.589896 1787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2612b6361ef24553a24ad71078591d4d" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec718ec869445058edcf766f03b5121"
I20260926 20:15:52.590031 1649 raft_consensus.cc:3037] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:52.590045 1787 raft_consensus.cc:3037] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:52.591118 1649 raft_consensus.cc:2454] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 1.
I20260926 20:15:52.591243 1787 raft_consensus.cc:2454] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 1.
I20260926 20:15:52.591414 1860 leader_election.cc:302] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac79daf846b410091bc714297b793ee, 6a1173b942ea4d4897ea419bac852baa; no voters:
I20260926 20:15:52.591601 2005 raft_consensus.cc:2781] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:52.591702 2005 raft_consensus.cc:686] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [term 1 LEADER]: Becoming Leader. State: Replica: 0ac79daf846b410091bc714297b793ee, State: Running, Role: LEADER
I20260926 20:15:52.591846 2005 consensus_queue.cc:237] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.592557 2008 tablet_replica.cc:439] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: TxnStatusTablet state changed. Reason: New leader 0ac79daf846b410091bc714297b793ee. Latest consensus state: current_term: 1 leader_uuid: "0ac79daf846b410091bc714297b793ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } } }
I20260926 20:15:52.592660 2008 tablet_replica.cc:442] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:15:52.592905 2020 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:15:52.593145 2020 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:15:52.593763 1488 catalog_manager.cc:5495] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac79daf846b410091bc714297b793ee (127.31.119.195). New cstate: current_term: 1 leader_uuid: "0ac79daf846b410091bc714297b793ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } health_report { overall_health: HEALTHY } } }
I20260926 20:15:52.619068 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.615139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.31.119.195:36951)
0926 20:15:52.615387 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.615406 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.615421 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:15:52.615496 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.615498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.615549 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.615668 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:52.615953 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.616499 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.617393 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.617651 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.617765 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.618075 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:52.618560 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:52.618767 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.618812 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.618880 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260926 20:15:52.619046 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.615013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:36951 (local address 127.0.0.1:50026)
0926 20:15:52.615175 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.615193 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.615324 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.615679 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.615682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.615742 (+ 60us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:15:52.615847 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.615852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.616612 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.616616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.617241 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.617250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.617767 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.617771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.617870 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.617916 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:15:52.618704 (+ 788us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:15:52.618721 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.618795 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.618895 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 20:15:52.624444 1787 raft_consensus.cc:1261] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Refusing update from remote peer 0ac79daf846b410091bc714297b793ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:52.624444 1649 raft_consensus.cc:1261] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Refusing update from remote peer 0ac79daf846b410091bc714297b793ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:52.625308 2008 consensus_queue.cc:1035] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:52.625507 2005 consensus_queue.cc:1035] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:52.629705 2004 tablet_replica.cc:439] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: TxnStatusTablet state changed. Reason: New leader 0ac79daf846b410091bc714297b793ee. Latest consensus state: current_term: 1 leader_uuid: "0ac79daf846b410091bc714297b793ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } } }
I20260926 20:15:52.629864 2004 tablet_replica.cc:442] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:52.631306 2006 tablet_replica.cc:439] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: TxnStatusTablet state changed. Reason: New leader 0ac79daf846b410091bc714297b793ee. Latest consensus state: current_term: 1 leader_uuid: "0ac79daf846b410091bc714297b793ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } } }
I20260926 20:15:52.631425 2006 tablet_replica.cc:442] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:52.632757 2008 tablet_replica.cc:439] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ac79daf846b410091bc714297b793ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } } }
I20260926 20:15:52.632891 2008 tablet_replica.cc:442] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:15:52.632907 2024 mvcc.cc:205] Tried to move back new op lower bound from 7333698570742607872 to 7333698570617700352. Current Snapshot: MvccSnapshot[applied={T|T < 7333698570742607872}]
I20260926 20:15:52.633011 2008 tablet_replica.cc:439] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ac79daf846b410091bc714297b793ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } } }
I20260926 20:15:52.633081 2008 tablet_replica.cc:442] T 2612b6361ef24553a24ad71078591d4d P 0ac79daf846b410091bc714297b793ee: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:15:52.633955 2004 tablet_replica.cc:439] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ac79daf846b410091bc714297b793ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } } }
I20260926 20:15:52.634068 2004 tablet_replica.cc:442] T 2612b6361ef24553a24ad71078591d4d P 7ec718ec869445058edcf766f03b5121: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:52.634493 2025 mvcc.cc:205] Tried to move back new op lower bound from 7333698570742607872 to 7333698570617700352. Current Snapshot: MvccSnapshot[applied={T|T < 7333698570742607872}]
I20260926 20:15:52.635337 2006 tablet_replica.cc:439] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ac79daf846b410091bc714297b793ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } } }
I20260926 20:15:52.635461 2006 tablet_replica.cc:442] T 2612b6361ef24553a24ad71078591d4d P 6a1173b942ea4d4897ea419bac852baa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:15:52.637172 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333698570742607872 to 7333698570617700352. Current Snapshot: MvccSnapshot[applied={T|T < 7333698570742607872}]
I20260926 20:15:52.646499 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.642753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.31.119.195:36951)
0926 20:15:52.642913 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.642918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.643098 (+ 180us) server_negotiation.cc:407] Connection header received
0926 20:15:52.643195 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.643198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.643247 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.643326 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:52.643593 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.644380 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.645264 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.645648 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.645755 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.645984 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:52.646076 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:52.646158 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.646301 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.646357 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 20:15:52.657945 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.654493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:35667 (local address 127.31.119.195:36951)
0926 20:15:52.654632 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.654637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.654812 (+ 175us) server_negotiation.cc:407] Connection header received
0926 20:15:52.654960 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.654962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.655007 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.655078 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:52.655369 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.655857 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.656810 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.657026 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.657128 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.657469 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:52.657541 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:52.657631 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.657715 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.657779 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":59}
I20260926 20:15:52.657953 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.654400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:36951 (local address 127.31.119.195:35667)
0926 20:15:52.654739 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.654751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.654865 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.655084 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.655086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.655092 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:15:52.655277 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.655282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.655962 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.655965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.656699 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.656708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.657133 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.657136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.657221 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.657266 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:15:52.657640 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:15:52.657647 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.657709 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.657807 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":52,"thread_start_us":84,"threads_started":1}
I20260926 20:15:52.664824 1787 raft_consensus.cc:1261] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Refusing update from remote peer 0ac79daf846b410091bc714297b793ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:52.664845 1649 raft_consensus.cc:1261] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Refusing update from remote peer 0ac79daf846b410091bc714297b793ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:52.665220 2029 consensus_queue.cc:1035] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:52.665427 2005 consensus_queue.cc:1035] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:52.708511 1491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:52.708770 1491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:52.711443 1491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144: Sending DeleteTablet for 3 replicas of tablet 1290352da8194162b556692df971fe09
I20260926 20:15:52.712229 1769 tablet_service.cc:1514] Processing DeleteTablet for tablet 1290352da8194162b556692df971fe09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:52 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:54424
I20260926 20:15:52.712476 1629 tablet_service.cc:1514] Processing DeleteTablet for tablet 1290352da8194162b556692df971fe09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:52 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:38592
I20260926 20:15:52.712847 2043 tablet_replica.cc:330] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: stopping tablet replica
I20260926 20:15:52.712940 2044 tablet_replica.cc:330] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: stopping tablet replica
I20260926 20:15:52.712946 1909 tablet_service.cc:1514] Processing DeleteTablet for tablet 1290352da8194162b556692df971fe09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:52 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:50024
I20260926 20:15:52.713236 2043 raft_consensus.cc:2229] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:52.713235 2044 raft_consensus.cc:2229] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:52.713500 2043 raft_consensus.cc:2258] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:52.713500 2044 raft_consensus.cc:2258] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:52.713853 1491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:15:52.714337 1491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:15:52.715111 2043 ts_tablet_manager.cc:1905] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:52.715111 2044 ts_tablet_manager.cc:1905] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:52.718389 2043 ts_tablet_manager.cc:1918] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:15:52.718590 2043 log.cc:1196] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/1290352da8194162b556692df971fe09
I20260926 20:15:52.718603 2044 ts_tablet_manager.cc:1918] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:15:52.718696 2044 log.cc:1196] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/1290352da8194162b556692df971fe09
I20260926 20:15:52.719059 2044 ts_tablet_manager.cc:1939] T 1290352da8194162b556692df971fe09 P 6a1173b942ea4d4897ea419bac852baa: Deleting consensus metadata
I20260926 20:15:52.719094 2043 ts_tablet_manager.cc:1939] T 1290352da8194162b556692df971fe09 P 7ec718ec869445058edcf766f03b5121: Deleting consensus metadata
I20260926 20:15:52.719978 1769 tablet_service.cc:1467] Processing CreateTablet for tablet b7f6dba392bc4ba2b239cdda1cc102f4 (DEFAULT_TABLE table=test-table [id=8b04dc115eb642e1b0d254e1cfe83fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:52.720274 1769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f6dba392bc4ba2b239cdda1cc102f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:52.720517 1478 catalog_manager.cc:4841] TS 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329): tablet 1290352da8194162b556692df971fe09 (table test-table [id=e6ca835b957842279d558c77d2368519]) successfully deleted
I20260926 20:15:52.720871 1476 catalog_manager.cc:4841] TS 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245): tablet 1290352da8194162b556692df971fe09 (table test-table [id=e6ca835b957842279d558c77d2368519]) successfully deleted
I20260926 20:15:52.721175 1629 tablet_service.cc:1467] Processing CreateTablet for tablet b7f6dba392bc4ba2b239cdda1cc102f4 (DEFAULT_TABLE table=test-table [id=8b04dc115eb642e1b0d254e1cfe83fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:52.721498 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f6dba392bc4ba2b239cdda1cc102f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:52.721839 2045 tablet_replica.cc:330] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: stopping tablet replica
I20260926 20:15:52.722359 2045 raft_consensus.cc:2229] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:15:52.722793 2045 raft_consensus.cc:2258] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:52.723385 1998 tablet_bootstrap.cc:492] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121: Bootstrap starting.
I20260926 20:15:52.723999 1909 tablet_service.cc:1467] Processing CreateTablet for tablet b7f6dba392bc4ba2b239cdda1cc102f4 (DEFAULT_TABLE table=test-table [id=8b04dc115eb642e1b0d254e1cfe83fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:52.724112 2000 tablet_bootstrap.cc:492] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa: Bootstrap starting.
I20260926 20:15:52.724328 1909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f6dba392bc4ba2b239cdda1cc102f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:52.724433 1998 tablet_bootstrap.cc:654] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:52.725052 2000 tablet_bootstrap.cc:654] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:52.726673 1998 tablet_bootstrap.cc:492] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121: No bootstrap required, opened a new log
I20260926 20:15:52.726758 1998 ts_tablet_manager.cc:1397] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 20:15:52.726904 2000 tablet_bootstrap.cc:492] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa: No bootstrap required, opened a new log
I20260926 20:15:52.726989 2000 ts_tablet_manager.cc:1397] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:52.727211 1999 tablet_bootstrap.cc:492] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee: Bootstrap starting.
I20260926 20:15:52.727262 1998 raft_consensus.cc:348] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.727375 1998 raft_consensus.cc:374] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:52.727406 1998 raft_consensus.cc:729] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec718ec869445058edcf766f03b5121, State: Initialized, Role: FOLLOWER
I20260926 20:15:52.727415 2000 raft_consensus.cc:348] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.727520 2000 raft_consensus.cc:374] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:52.727550 2000 raft_consensus.cc:729] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1173b942ea4d4897ea419bac852baa, State: Initialized, Role: FOLLOWER
I20260926 20:15:52.727533 1998 consensus_queue.cc:260] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.727702 2000 consensus_queue.cc:260] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.727975 2000 ts_tablet_manager.cc:1428] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:15:52.728206 1999 tablet_bootstrap.cc:654] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:52.730288 1998 ts_tablet_manager.cc:1428] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:15:52.730284 2045 ts_tablet_manager.cc:1905] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:52.733443 2045 ts_tablet_manager.cc:1918] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:15:52.733560 1999 tablet_bootstrap.cc:492] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee: No bootstrap required, opened a new log
I20260926 20:15:52.733560 2045 log.cc:1196] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/1290352da8194162b556692df971fe09
I20260926 20:15:52.733700 1999 ts_tablet_manager.cc:1397] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 20:15:52.733989 2045 ts_tablet_manager.cc:1939] T 1290352da8194162b556692df971fe09 P 0ac79daf846b410091bc714297b793ee: Deleting consensus metadata
I20260926 20:15:52.734210 1999 raft_consensus.cc:348] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.734340 1999 raft_consensus.cc:374] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:52.734385 1999 raft_consensus.cc:729] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac79daf846b410091bc714297b793ee, State: Initialized, Role: FOLLOWER
I20260926 20:15:52.734520 1999 consensus_queue.cc:260] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.734843 1999 ts_tablet_manager.cc:1428] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:15:52.735214 1479 catalog_manager.cc:4841] TS 0ac79daf846b410091bc714297b793ee (127.31.119.195:36951): tablet 1290352da8194162b556692df971fe09 (table test-table [id=e6ca835b957842279d558c77d2368519]) successfully deleted
I20260926 20:15:52.888341 2004 raft_consensus.cc:482] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:15:52.888509 2004 raft_consensus.cc:504] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.889812 2004 leader_election.cc:288] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245), 0ac79daf846b410091bc714297b793ee (127.31.119.195:36951)
I20260926 20:15:52.894771 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.890491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:55103 (local address 127.31.119.195:36951)
0926 20:15:52.890612 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.890616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.890633 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:15:52.890714 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.890717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.890772 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.890852 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:52.891206 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.891966 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.893724 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.894110 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.894265 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.894324 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.894529 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.894624 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:15:52.894838 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.889933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40245 (local address 127.31.119.194:59699)
0926 20:15:52.890438 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.890452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.890616 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.891015 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.891018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.891031 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:52.891108 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.891118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.891966 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.891968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.893562 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.893572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.894435 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.894441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.894538 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.894556 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.894623 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.894689 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":197,"thread_start_us":158,"threads_started":1}
I20260926 20:15:52.894873 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.890136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:59699 (local address 127.31.119.193:40245)
0926 20:15:52.890290 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.890296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.890527 (+ 231us) server_negotiation.cc:407] Connection header received
0926 20:15:52.890744 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.890747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.890800 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.890848 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:15:52.891276 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.891797 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.893670 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.894317 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.894458 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.894509 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.894657 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.894742 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:15:52.895360 1649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f6dba392bc4ba2b239cdda1cc102f4" candidate_uuid: "7ec718ec869445058edcf766f03b5121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1173b942ea4d4897ea419bac852baa" is_pre_election: true
I20260926 20:15:52.895531 1649 raft_consensus.cc:2454] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec718ec869445058edcf766f03b5121 in term 0.
I20260926 20:15:52.895644 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.890074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:36951 (local address 127.31.119.194:55103)
0926 20:15:52.890527 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.890541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.890616 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.890903 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.890905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.890924 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:15:52.891036 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.891043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.892096 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.892099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.893561 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.893572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.894265 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.894269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.894360 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.894410 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.895405 (+ 995us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.895485 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":11,"spinlock_wait_cycles":4864,"thread_start_us":119,"threads_started":1}
I20260926 20:15:52.895946 1720 leader_election.cc:302] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1173b942ea4d4897ea419bac852baa, 7ec718ec869445058edcf766f03b5121; no voters:
I20260926 20:15:52.896142 1929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f6dba392bc4ba2b239cdda1cc102f4" candidate_uuid: "7ec718ec869445058edcf766f03b5121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac79daf846b410091bc714297b793ee" is_pre_election: true
I20260926 20:15:52.896238 2004 raft_consensus.cc:2781] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:15:52.896302 2004 raft_consensus.cc:482] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:15:52.896341 2004 raft_consensus.cc:3037] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:52.896365 1929 raft_consensus.cc:2454] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec718ec869445058edcf766f03b5121 in term 0.
I20260926 20:15:52.898005 2004 raft_consensus.cc:504] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.898393 2004 leader_election.cc:288] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [CANDIDATE]: Term 1 election: Requested vote from peers 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245), 0ac79daf846b410091bc714297b793ee (127.31.119.195:36951)
I20260926 20:15:52.898762 1649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f6dba392bc4ba2b239cdda1cc102f4" candidate_uuid: "7ec718ec869445058edcf766f03b5121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1173b942ea4d4897ea419bac852baa"
I20260926 20:15:52.898911 1649 raft_consensus.cc:3037] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:52.898881 1929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f6dba392bc4ba2b239cdda1cc102f4" candidate_uuid: "7ec718ec869445058edcf766f03b5121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac79daf846b410091bc714297b793ee"
I20260926 20:15:52.898990 1929 raft_consensus.cc:3037] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:52.900147 1649 raft_consensus.cc:2454] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec718ec869445058edcf766f03b5121 in term 1.
I20260926 20:15:52.900247 1929 raft_consensus.cc:2454] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec718ec869445058edcf766f03b5121 in term 1.
I20260926 20:15:52.900599 1723 leader_election.cc:302] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac79daf846b410091bc714297b793ee, 7ec718ec869445058edcf766f03b5121; no voters:
I20260926 20:15:52.900863 2004 raft_consensus.cc:2781] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:52.901049 2004 raft_consensus.cc:686] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec718ec869445058edcf766f03b5121, State: Running, Role: LEADER
I20260926 20:15:52.901357 2004 consensus_queue.cc:237] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:52.903797 1488 catalog_manager.cc:5495] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ec718ec869445058edcf766f03b5121 (127.31.119.194). New cstate: current_term: 1 leader_uuid: "7ec718ec869445058edcf766f03b5121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } health_report { overall_health: UNKNOWN } } }
I20260926 20:15:52.935424 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.931627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.31.119.194:37329)
0926 20:15:52.931929 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.931939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.931953 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:15:52.932035 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.932037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.932089 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.932235 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:52.932464 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.932978 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.933890 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.934116 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.934231 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.934493 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:52.935000 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:52.935127 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.935171 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.935226 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":115,"threads_started":1}
I20260926 20:15:52.947235 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.943665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:36951 (local address 127.31.119.194:55811)
0926 20:15:52.943959 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.943976 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.944088 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.944360 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.944363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.944371 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:15:52.944502 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.944507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.945233 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.945239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.945974 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.945984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.946444 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.946447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.946551 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.946599 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:15:52.946916 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:15:52.946926 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.946990 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.947075 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":106,"threads_started":1}
I20260926 20:15:52.947221 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.943892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:55811 (local address 127.31.119.195:36951)
0926 20:15:52.944032 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.944037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.944053 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:15:52.944217 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.944220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.944269 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.944349 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:52.944610 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.945108 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.946100 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.946330 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.946440 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.946753 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:52.946832 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:52.946910 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.947001 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.947056 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:15:52.953588 1649 raft_consensus.cc:1261] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Refusing update from remote peer 7ec718ec869445058edcf766f03b5121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:52.953878 1929 raft_consensus.cc:1261] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Refusing update from remote peer 7ec718ec869445058edcf766f03b5121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:52.954059 2004 consensus_queue.cc:1035] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:52.954429 2051 consensus_queue.cc:1035] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:52.971966 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.968714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:37471 (local address 127.31.119.194:37329)
0926 20:15:52.968832 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:52.968835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:52.968870 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:15:52.969023 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:52.969027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:52.969077 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:52.969167 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:52.969443 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.970022 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.970854 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.971110 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.971224 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.971508 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:52.971593 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:52.971668 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:52.971771 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 20:15:52.971827 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:15:52.971966 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:52.968662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:37329 (local address 127.31.119.195:37471)
0926 20:15:52.968811 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:52.968828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:52.968920 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:52.969185 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:52.969188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:52.969196 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:15:52.969331 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.969338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.970133 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.970138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.970736 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:52.970745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:52.971222 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:52.971225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:52.971339 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:52.971376 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:15:52.971676 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:15:52.971681 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:15:52.971755 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:15:52.971850 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:15:52.972803 2026 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:15:54.534514 1492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:54.534724 1492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:54.537531 1492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144: Sending DeleteTablet for 3 replicas of tablet b7f6dba392bc4ba2b239cdda1cc102f4
I20260926 20:15:54.538149 1629 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f6dba392bc4ba2b239cdda1cc102f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:54 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:38592
I20260926 20:15:54.538246 1769 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f6dba392bc4ba2b239cdda1cc102f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:54 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:54424
I20260926 20:15:54.538486 2082 tablet_replica.cc:330] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa: stopping tablet replica
I20260926 20:15:54.538621 2082 raft_consensus.cc:2229] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:54.538738 2082 raft_consensus.cc:2258] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:54.541935 2082 ts_tablet_manager.cc:1905] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:54.542213 1908 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f6dba392bc4ba2b239cdda1cc102f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:54 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:50024
I20260926 20:15:54.542593 2085 tablet_replica.cc:330] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee: stopping tablet replica
I20260926 20:15:54.542640 2083 tablet_replica.cc:330] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121: stopping tablet replica
I20260926 20:15:54.542798 2083 raft_consensus.cc:2229] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:15:54.542800 2085 raft_consensus.cc:2229] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:54.542934 2085 raft_consensus.cc:2258] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:54.543167 2083 raft_consensus.cc:2258] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:54.543630 2085 ts_tablet_manager.cc:1905] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:54.543813 2083 ts_tablet_manager.cc:1905] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:54.544749 2082 ts_tablet_manager.cc:1918] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:15:54.544843 2082 log.cc:1196] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/b7f6dba392bc4ba2b239cdda1cc102f4
I20260926 20:15:54.546330 2083 ts_tablet_manager.cc:1918] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:15:54.546330 2085 ts_tablet_manager.cc:1918] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:15:54.546413 2085 log.cc:1196] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/b7f6dba392bc4ba2b239cdda1cc102f4
I20260926 20:15:54.546542 2083 log.cc:1196] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/b7f6dba392bc4ba2b239cdda1cc102f4
I20260926 20:15:54.546711 2085 ts_tablet_manager.cc:1939] T b7f6dba392bc4ba2b239cdda1cc102f4 P 0ac79daf846b410091bc714297b793ee: Deleting consensus metadata
I20260926 20:15:54.546860 2083 ts_tablet_manager.cc:1939] T b7f6dba392bc4ba2b239cdda1cc102f4 P 7ec718ec869445058edcf766f03b5121: Deleting consensus metadata
I20260926 20:15:54.547500 1478 catalog_manager.cc:4841] TS 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329): tablet b7f6dba392bc4ba2b239cdda1cc102f4 (table test-table [id=8b04dc115eb642e1b0d254e1cfe83fa0]) successfully deleted
I20260926 20:15:54.547758 1479 catalog_manager.cc:4841] TS 0ac79daf846b410091bc714297b793ee (127.31.119.195:36951): tablet b7f6dba392bc4ba2b239cdda1cc102f4 (table test-table [id=8b04dc115eb642e1b0d254e1cfe83fa0]) successfully deleted
I20260926 20:15:54.549742 2082 ts_tablet_manager.cc:1939] T b7f6dba392bc4ba2b239cdda1cc102f4 P 6a1173b942ea4d4897ea419bac852baa: Deleting consensus metadata
I20260926 20:15:54.550633 1476 catalog_manager.cc:4841] TS 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245): tablet b7f6dba392bc4ba2b239cdda1cc102f4 (table test-table [id=8b04dc115eb642e1b0d254e1cfe83fa0]) successfully deleted
Sep 26 20:15:54 dist-test-slave-vnbh krb5kdc[1447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453754, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:15:54 dist-test-slave-vnbh krb5kdc[1447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453754, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:54.587034 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:54.568874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39612 (local address 127.31.119.254:36659)
0926 20:15:54.569220 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:54.569227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:54.569341 (+ 114us) server_negotiation.cc:407] Connection header received
0926 20:15:54.569529 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:54.569532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:54.569611 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:54.569733 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:54.570696 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:54.571581 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:54.572882 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:54.573257 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:54.573400 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:54.583380 (+ 9980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:54.583384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:54.583386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:54.583423 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:54.585158 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:54.585815 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:54.585818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:54.585819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:54.585913 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:54.586205 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:54.586208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:54.586209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:54.586419 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:54.586526 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:54.586773 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 20:15:54.586895 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":122,"threads_started":1}
I20260926 20:15:54.590197 1492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:15:54.590759 1492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:15:54.596408 1629 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3bfafaef5e4373b7e00c6b008eb3ba (DEFAULT_TABLE table=test-table [id=f27ae1ab2fa14ce58ce37948c16b47bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:54.596481 1769 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3bfafaef5e4373b7e00c6b008eb3ba (DEFAULT_TABLE table=test-table [id=f27ae1ab2fa14ce58ce37948c16b47bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:54.596572 1908 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3bfafaef5e4373b7e00c6b008eb3ba (DEFAULT_TABLE table=test-table [id=f27ae1ab2fa14ce58ce37948c16b47bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:54.596751 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3bfafaef5e4373b7e00c6b008eb3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:54.596850 1908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3bfafaef5e4373b7e00c6b008eb3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:54.596849 1769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3bfafaef5e4373b7e00c6b008eb3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:54.598907 2093 tablet_bootstrap.cc:492] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa: Bootstrap starting.
I20260926 20:15:54.599591 2095 tablet_bootstrap.cc:492] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee: Bootstrap starting.
I20260926 20:15:54.600268 2093 tablet_bootstrap.cc:654] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:54.600401 2095 tablet_bootstrap.cc:654] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:54.601042 2094 tablet_bootstrap.cc:492] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121: Bootstrap starting.
I20260926 20:15:54.601975 2094 tablet_bootstrap.cc:654] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:54.601977 2093 tablet_bootstrap.cc:492] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa: No bootstrap required, opened a new log
I20260926 20:15:54.602128 2093 ts_tablet_manager.cc:1397] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:54.602675 2093 raft_consensus.cc:348] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.602823 2093 raft_consensus.cc:374] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:54.602862 2093 raft_consensus.cc:729] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1173b942ea4d4897ea419bac852baa, State: Initialized, Role: FOLLOWER
I20260926 20:15:54.602895 2095 tablet_bootstrap.cc:492] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee: No bootstrap required, opened a new log
I20260926 20:15:54.603006 2095 ts_tablet_manager.cc:1397] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:54.603075 2093 consensus_queue.cc:260] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.603273 2094 tablet_bootstrap.cc:492] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121: No bootstrap required, opened a new log
I20260926 20:15:54.603401 2094 ts_tablet_manager.cc:1397] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:15:54.603443 2095 raft_consensus.cc:348] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.603498 2093 ts_tablet_manager.cc:1428] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:15:54.603569 2095 raft_consensus.cc:374] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:54.603614 2095 raft_consensus.cc:729] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac79daf846b410091bc714297b793ee, State: Initialized, Role: FOLLOWER
I20260926 20:15:54.603751 2095 consensus_queue.cc:260] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.603861 2094 raft_consensus.cc:348] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.603971 2094 raft_consensus.cc:374] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:54.604001 2094 raft_consensus.cc:729] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec718ec869445058edcf766f03b5121, State: Initialized, Role: FOLLOWER
I20260926 20:15:54.604127 2094 consensus_queue.cc:260] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.604496 2094 ts_tablet_manager.cc:1428] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:15:54.605744 2095 ts_tablet_manager.cc:1428] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:15:54.765455 2063 raft_consensus.cc:482] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:15:54.765722 2063 raft_consensus.cc:504] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.766273 2063 leader_election.cc:288] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245), 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329)
I20260926 20:15:54.766721 1649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3bfafaef5e4373b7e00c6b008eb3ba" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1173b942ea4d4897ea419bac852baa" is_pre_election: true
I20260926 20:15:54.766785 1787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3bfafaef5e4373b7e00c6b008eb3ba" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec718ec869445058edcf766f03b5121" is_pre_election: true
I20260926 20:15:54.766880 1649 raft_consensus.cc:2454] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 0.
I20260926 20:15:54.766968 1787 raft_consensus.cc:2454] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 0.
I20260926 20:15:54.767256 1860 leader_election.cc:302] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac79daf846b410091bc714297b793ee, 6a1173b942ea4d4897ea419bac852baa; no voters:
I20260926 20:15:54.767439 2063 raft_consensus.cc:2781] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:15:54.767498 2063 raft_consensus.cc:482] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:15:54.767525 2063 raft_consensus.cc:3037] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:54.768561 2063 raft_consensus.cc:504] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.768931 2063 leader_election.cc:288] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 election: Requested vote from peers 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245), 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329)
I20260926 20:15:54.769397 1787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3bfafaef5e4373b7e00c6b008eb3ba" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec718ec869445058edcf766f03b5121"
I20260926 20:15:54.769394 1649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3bfafaef5e4373b7e00c6b008eb3ba" candidate_uuid: "0ac79daf846b410091bc714297b793ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1173b942ea4d4897ea419bac852baa"
I20260926 20:15:54.769515 1787 raft_consensus.cc:3037] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:54.769542 1649 raft_consensus.cc:3037] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:54.770558 1649 raft_consensus.cc:2454] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 1.
I20260926 20:15:54.770673 1787 raft_consensus.cc:2454] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac79daf846b410091bc714297b793ee in term 1.
I20260926 20:15:54.770939 1860 leader_election.cc:302] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac79daf846b410091bc714297b793ee, 6a1173b942ea4d4897ea419bac852baa; no voters:
I20260926 20:15:54.771124 2063 raft_consensus.cc:2781] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:54.771237 2063 raft_consensus.cc:686] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 1 LEADER]: Becoming Leader. State: Replica: 0ac79daf846b410091bc714297b793ee, State: Running, Role: LEADER
I20260926 20:15:54.771386 2063 consensus_queue.cc:237] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.773120 1492 catalog_manager.cc:5495] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac79daf846b410091bc714297b793ee (127.31.119.195). New cstate: current_term: 1 leader_uuid: "0ac79daf846b410091bc714297b793ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } health_report { overall_health: HEALTHY } } }
I20260926 20:15:54.805224 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:54.801927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.31.119.195:36951)
0926 20:15:54.802203 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:54.802211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:54.802225 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:15:54.802265 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:54.802267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:54.802309 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:54.802381 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:54.802665 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:54.803234 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:54.804218 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:54.804436 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:54.804515 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:54.804784 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:54.804850 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:54.804902 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:54.804991 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 20:15:54.805034 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":154,"threads_started":1}
I20260926 20:15:54.811237 1787 raft_consensus.cc:1261] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Refusing update from remote peer 0ac79daf846b410091bc714297b793ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:54.811285 1649 raft_consensus.cc:1261] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Refusing update from remote peer 0ac79daf846b410091bc714297b793ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:54.811594 2068 consensus_queue.cc:1035] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:54.811791 2029 consensus_queue.cc:1035] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:54.843853 1491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:54.844007 1491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:54.846246 1491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144: Sending DeleteTablet for 3 replicas of tablet 9e3bfafaef5e4373b7e00c6b008eb3ba
I20260926 20:15:54.846917 1908 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3bfafaef5e4373b7e00c6b008eb3ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:54 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:50024
I20260926 20:15:54.846976 1769 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3bfafaef5e4373b7e00c6b008eb3ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:54 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:54424
I20260926 20:15:54.846989 1629 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3bfafaef5e4373b7e00c6b008eb3ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:54 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:38592
I20260926 20:15:54.847115 2085 tablet_replica.cc:330] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee: stopping tablet replica
I20260926 20:15:54.847165 2082 tablet_replica.cc:330] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa: stopping tablet replica
I20260926 20:15:54.847165 2083 tablet_replica.cc:330] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121: stopping tablet replica
I20260926 20:15:54.847241 2085 raft_consensus.cc:2229] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:15:54.847270 2082 raft_consensus.cc:2229] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:54.847276 2083 raft_consensus.cc:2229] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:54.847388 2082 raft_consensus.cc:2258] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:54.847388 2083 raft_consensus.cc:2258] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:54.847436 2085 raft_consensus.cc:2258] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:54.848240 2082 ts_tablet_manager.cc:1905] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:54.848272 2083 ts_tablet_manager.cc:1905] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:54.848222 1491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:15:54.848517 2085 ts_tablet_manager.cc:1905] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:15:54.848671 1491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:15:54.850878 2083 ts_tablet_manager.cc:1918] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:15:54.850878 2082 ts_tablet_manager.cc:1918] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:15:54.850953 2082 log.cc:1196] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/9e3bfafaef5e4373b7e00c6b008eb3ba
I20260926 20:15:54.850953 2083 log.cc:1196] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/9e3bfafaef5e4373b7e00c6b008eb3ba
I20260926 20:15:54.851226 2082 ts_tablet_manager.cc:1939] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 6a1173b942ea4d4897ea419bac852baa: Deleting consensus metadata
I20260926 20:15:54.851234 2083 ts_tablet_manager.cc:1939] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 7ec718ec869445058edcf766f03b5121: Deleting consensus metadata
I20260926 20:15:54.851279 2085 ts_tablet_manager.cc:1918] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:15:54.851352 2085 log.cc:1196] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/9e3bfafaef5e4373b7e00c6b008eb3ba
I20260926 20:15:54.851612 2085 ts_tablet_manager.cc:1939] T 9e3bfafaef5e4373b7e00c6b008eb3ba P 0ac79daf846b410091bc714297b793ee: Deleting consensus metadata
I20260926 20:15:54.852212 1476 catalog_manager.cc:4841] TS 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245): tablet 9e3bfafaef5e4373b7e00c6b008eb3ba (table test-table [id=f27ae1ab2fa14ce58ce37948c16b47bc]) successfully deleted
I20260926 20:15:54.852212 1478 catalog_manager.cc:4841] TS 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329): tablet 9e3bfafaef5e4373b7e00c6b008eb3ba (table test-table [id=f27ae1ab2fa14ce58ce37948c16b47bc]) successfully deleted
I20260926 20:15:54.852448 1479 catalog_manager.cc:4841] TS 0ac79daf846b410091bc714297b793ee (127.31.119.195:36951): tablet 9e3bfafaef5e4373b7e00c6b008eb3ba (table test-table [id=f27ae1ab2fa14ce58ce37948c16b47bc]) successfully deleted
I20260926 20:15:54.853966 1629 tablet_service.cc:1467] Processing CreateTablet for tablet ee18594b6f8541b19c8b5ddee5a4b05e (DEFAULT_TABLE table=test-table [id=a0b7b3de23864c80bf6716c5646f0919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:54.854305 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee18594b6f8541b19c8b5ddee5a4b05e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:54.854246 1769 tablet_service.cc:1467] Processing CreateTablet for tablet ee18594b6f8541b19c8b5ddee5a4b05e (DEFAULT_TABLE table=test-table [id=a0b7b3de23864c80bf6716c5646f0919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:54.854491 1769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee18594b6f8541b19c8b5ddee5a4b05e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:54.856851 2093 tablet_bootstrap.cc:492] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa: Bootstrap starting.
I20260926 20:15:54.856851 2094 tablet_bootstrap.cc:492] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121: Bootstrap starting.
I20260926 20:15:54.857331 1908 tablet_service.cc:1467] Processing CreateTablet for tablet ee18594b6f8541b19c8b5ddee5a4b05e (DEFAULT_TABLE table=test-table [id=a0b7b3de23864c80bf6716c5646f0919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:15:54.857630 1908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee18594b6f8541b19c8b5ddee5a4b05e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:54.857811 2093 tablet_bootstrap.cc:654] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:54.857811 2094 tablet_bootstrap.cc:654] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:54.859163 2094 tablet_bootstrap.cc:492] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121: No bootstrap required, opened a new log
I20260926 20:15:54.859282 2094 ts_tablet_manager.cc:1397] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:15:54.859292 2093 tablet_bootstrap.cc:492] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa: No bootstrap required, opened a new log
I20260926 20:15:54.859370 2093 ts_tablet_manager.cc:1397] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 20:15:54.859767 2094 raft_consensus.cc:348] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.859884 2094 raft_consensus.cc:374] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:54.859923 2094 raft_consensus.cc:729] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec718ec869445058edcf766f03b5121, State: Initialized, Role: FOLLOWER
I20260926 20:15:54.859905 2093 raft_consensus.cc:348] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.860016 2095 tablet_bootstrap.cc:492] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee: Bootstrap starting.
I20260926 20:15:54.860026 2093 raft_consensus.cc:374] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:54.860060 2093 raft_consensus.cc:729] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1173b942ea4d4897ea419bac852baa, State: Initialized, Role: FOLLOWER
I20260926 20:15:54.860059 2094 consensus_queue.cc:260] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.860177 2093 consensus_queue.cc:260] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.860447 2093 ts_tablet_manager.cc:1428] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:15:54.860834 2095 tablet_bootstrap.cc:654] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:54.861570 2094 ts_tablet_manager.cc:1428] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:15:54.862718 2095 tablet_bootstrap.cc:492] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee: No bootstrap required, opened a new log
I20260926 20:15:54.862834 2095 ts_tablet_manager.cc:1397] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:15:54.863279 2095 raft_consensus.cc:348] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.863399 2095 raft_consensus.cc:374] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:54.863440 2095 raft_consensus.cc:729] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac79daf846b410091bc714297b793ee, State: Initialized, Role: FOLLOWER
I20260926 20:15:54.863595 2095 consensus_queue.cc:260] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:54.863873 2095 ts_tablet_manager.cc:1428] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:15:55.005016 2073 raft_consensus.cc:482] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:15:55.005168 2073 raft_consensus.cc:504] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:55.005631 2073 leader_election.cc:288] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245), 0ac79daf846b410091bc714297b793ee (127.31.119.195:36951)
I20260926 20:15:55.006266 1649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee18594b6f8541b19c8b5ddee5a4b05e" candidate_uuid: "7ec718ec869445058edcf766f03b5121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1173b942ea4d4897ea419bac852baa" is_pre_election: true
I20260926 20:15:55.006266 1929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee18594b6f8541b19c8b5ddee5a4b05e" candidate_uuid: "7ec718ec869445058edcf766f03b5121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac79daf846b410091bc714297b793ee" is_pre_election: true
I20260926 20:15:55.006441 1929 raft_consensus.cc:2454] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec718ec869445058edcf766f03b5121 in term 0.
I20260926 20:15:55.006441 1649 raft_consensus.cc:2454] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec718ec869445058edcf766f03b5121 in term 0.
I20260926 20:15:55.006876 1723 leader_election.cc:302] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac79daf846b410091bc714297b793ee, 7ec718ec869445058edcf766f03b5121; no voters:
I20260926 20:15:55.007110 2073 raft_consensus.cc:2781] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:15:55.007174 2073 raft_consensus.cc:482] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:15:55.007227 2073 raft_consensus.cc:3037] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:55.008343 2073 raft_consensus.cc:504] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:55.008737 2073 leader_election.cc:288] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [CANDIDATE]: Term 1 election: Requested vote from peers 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245), 0ac79daf846b410091bc714297b793ee (127.31.119.195:36951)
I20260926 20:15:55.009246 1649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee18594b6f8541b19c8b5ddee5a4b05e" candidate_uuid: "7ec718ec869445058edcf766f03b5121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1173b942ea4d4897ea419bac852baa"
I20260926 20:15:55.009246 1929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee18594b6f8541b19c8b5ddee5a4b05e" candidate_uuid: "7ec718ec869445058edcf766f03b5121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac79daf846b410091bc714297b793ee"
I20260926 20:15:55.009411 1649 raft_consensus.cc:3037] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:55.009411 1929 raft_consensus.cc:3037] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:55.010615 1649 raft_consensus.cc:2454] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec718ec869445058edcf766f03b5121 in term 1.
I20260926 20:15:55.010612 1929 raft_consensus.cc:2454] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec718ec869445058edcf766f03b5121 in term 1.
I20260926 20:15:55.010965 1720 leader_election.cc:302] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1173b942ea4d4897ea419bac852baa, 7ec718ec869445058edcf766f03b5121; no voters:
I20260926 20:15:55.011165 2073 raft_consensus.cc:2781] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:55.011261 2073 raft_consensus.cc:686] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec718ec869445058edcf766f03b5121, State: Running, Role: LEADER
I20260926 20:15:55.011415 2073 consensus_queue.cc:237] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } }
I20260926 20:15:55.013154 1491 catalog_manager.cc:5495] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ec718ec869445058edcf766f03b5121 (127.31.119.194). New cstate: current_term: 1 leader_uuid: "7ec718ec869445058edcf766f03b5121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec718ec869445058edcf766f03b5121" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 } health_report { overall_health: UNKNOWN } } }
I20260926 20:15:55.063864 2111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:55.060342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.31.119.194:37329)
0926 20:15:55.060606 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:55.060624 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:55.060637 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:15:55.060678 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:55.060680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:55.060721 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:55.060794 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:55.061113 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:55.061748 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:55.062797 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:55.063032 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:55.063146 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:55.063448 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:55.063523 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:55.063590 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:55.063675 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:15:55.063725 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
I20260926 20:15:55.071225 1929 raft_consensus.cc:1261] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Refusing update from remote peer 7ec718ec869445058edcf766f03b5121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:55.071352 1649 raft_consensus.cc:1261] T ee18594b6f8541b19c8b5ddee5a4b05e P 6a1173b942ea4d4897ea419bac852baa [term 1 FOLLOWER]: Refusing update from remote peer 7ec718ec869445058edcf766f03b5121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:55.071713 2073 consensus_queue.cc:1035] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac79daf846b410091bc714297b793ee" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 36951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:55.071895 2004 consensus_queue.cc:1035] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1173b942ea4d4897ea419bac852baa" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:15:55.087699 2064 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:15:56.643018 1491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:56.643179 1491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:15:56.645931 1491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 224e23cad6554d83beed9171478b2144: Sending DeleteTablet for 3 replicas of tablet ee18594b6f8541b19c8b5ddee5a4b05e
I20260926 20:15:56.646566 1769 tablet_service.cc:1514] Processing DeleteTablet for tablet ee18594b6f8541b19c8b5ddee5a4b05e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:56 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:54424
I20260926 20:15:56.646566 1629 tablet_service.cc:1514] Processing DeleteTablet for tablet ee18594b6f8541b19c8b5ddee5a4b05e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:56 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:38592
I20260926 20:15:56.646715 1908 tablet_service.cc:1514] Processing DeleteTablet for tablet ee18594b6f8541b19c8b5ddee5a4b05e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:15:56 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:50024
I20260926 20:15:56.647040 2137 tablet_replica.cc:330] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121: stopping tablet replica
I20260926 20:15:56.647055 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 1564
I20260926 20:15:56.648566 2137 raft_consensus.cc:2229] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:15:56.648810 2137 raft_consensus.cc:2258] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:56.649124 2139 tablet_replica.cc:330] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee: stopping tablet replica
I20260926 20:15:56.649278 2139 raft_consensus.cc:2229] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:15:56.649529 2139 raft_consensus.cc:2258] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:15:56.649636 2137 ts_tablet_manager.cc:1905] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:56.650276 2139 ts_tablet_manager.cc:1905] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:15:56.652518 2139 ts_tablet_manager.cc:1918] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:15:56.652609 2139 log.cc:1196] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/ee18594b6f8541b19c8b5ddee5a4b05e
I20260926 20:15:56.652918 2139 ts_tablet_manager.cc:1939] T ee18594b6f8541b19c8b5ddee5a4b05e P 0ac79daf846b410091bc714297b793ee: Deleting consensus metadata
I20260926 20:15:56.653622 2137 ts_tablet_manager.cc:1918] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:15:56.653712 2137 log.cc:1196] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/ee18594b6f8541b19c8b5ddee5a4b05e
I20260926 20:15:56.653789 1479 catalog_manager.cc:4841] TS 0ac79daf846b410091bc714297b793ee (127.31.119.195:36951): tablet ee18594b6f8541b19c8b5ddee5a4b05e (table test-table [id=a0b7b3de23864c80bf6716c5646f0919]) successfully deleted
I20260926 20:15:56.654003 2137 ts_tablet_manager.cc:1939] T ee18594b6f8541b19c8b5ddee5a4b05e P 7ec718ec869445058edcf766f03b5121: Deleting consensus metadata
W20260926 20:15:56.654789 1476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:40245 (error 108)
I20260926 20:15:56.654851 1478 catalog_manager.cc:4841] TS 7ec718ec869445058edcf766f03b5121 (127.31.119.194:37329): tablet ee18594b6f8541b19c8b5ddee5a4b05e (table test-table [id=a0b7b3de23864c80bf6716c5646f0919]) successfully deleted
I20260926 20:15:56.655012 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 1704
I20260926 20:15:56.656560 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:56.656156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40245 (local address 127.0.0.1:49084)
0926 20:15:56.656393 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:56.656473 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:40245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
W20260926 20:15:56.656704 1476 catalog_manager.cc:4584] TS 6a1173b942ea4d4897ea419bac852baa (127.31.119.193:40245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee18594b6f8541b19c8b5ddee5a4b05e: Network error: Client connection negotiation failed: client connection to 127.31.119.193:40245: connect: Connection refused (error 111)
I20260926 20:15:56.662360 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 1844
I20260926 20:15:56.669474 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 1460
2026-09-26T20:15:56Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6832 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:15:56 dist-test-slave-vnbh krb5kdc[2144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:15:56 dist-test-slave-vnbh krb5kdc[2144](info): set up 2 sockets
Sep 26 20:15:56 dist-test-slave-vnbh krb5kdc[2144](info): commencing operation
krb5kdc: starting...
W20260926 20:15:57.722800 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:15:57 dist-test-slave-vnbh krb5kdc[2144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:15:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:15:57.853529 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:43261
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:43261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:57.966893 2157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:57.967198 2157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:57.967240 2157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:57.971485 2157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:57.971547 2157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:57.971567 2157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:57.971585 2157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:57.971599 2157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:57.977398 2157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:43261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:43261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:57.978577 2157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:57.979673 2157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:57.988983 2162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:57.989078 2163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:57.988996 2165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:57.989431 2157 server_base.cc:1034] running on GCE node
I20260926 20:15:57.990161 2157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:57.991448 2157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:57.992668 2157 hybrid_clock.cc:648] HybridClock initialized: now 1790453757992659 us; error 38 us; skew 500 ppm
Sep 26 20:15:57 dist-test-slave-vnbh krb5kdc[2144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453757, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:57.996044 2157 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:15:57.997109 2157 webserver.cc:460] Webserver started at http://127.31.119.254:38085/ using document root <none> and password file <none>
I20260926 20:15:57.998138 2157 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:57.998256 2157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:57.998500 2157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:58.000991 2157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "ba806585f81241b78beb15578fd9e11e"
format_stamp: "Formatted at 2026-09-26 20:15:57 on dist-test-slave-vnbh"
server_key: "c4dc4b5ddda2f004ecce5d72cf013c52"
server_key_iv: "dceda8ef793dd33eb682546a3e750c46"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.001724 2157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "ba806585f81241b78beb15578fd9e11e"
format_stamp: "Formatted at 2026-09-26 20:15:57 on dist-test-slave-vnbh"
server_key: "c4dc4b5ddda2f004ecce5d72cf013c52"
server_key_iv: "dceda8ef793dd33eb682546a3e750c46"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.006294 2157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:15:58.009240 2172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:58.010730 2157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:15:58.010901 2157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "ba806585f81241b78beb15578fd9e11e"
format_stamp: "Formatted at 2026-09-26 20:15:57 on dist-test-slave-vnbh"
server_key: "c4dc4b5ddda2f004ecce5d72cf013c52"
server_key_iv: "dceda8ef793dd33eb682546a3e750c46"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.011034 2157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:58.040047 2157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:58.043699 2157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:58.043948 2157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:58.051378 2157 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:43261
I20260926 20:15:58.051399 2224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:43261 every 8 connection(s)
I20260926 20:15:58.052395 2157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:15:58.055369 2225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:58.060748 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 2157
I20260926 20:15:58.060848 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:15:58.061136 32223 external_mini_cluster.cc:1428] Setting key eef66177f788da2ec6e47758e52b1678
I20260926 20:15:58.061618 2225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e: Bootstrap starting.
I20260926 20:15:58.064373 2225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:58.065214 2225 log.cc:824] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:58.067325 2225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e: No bootstrap required, opened a new log
Sep 26 20:15:58 dist-test-slave-vnbh krb5kdc[2144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:58.071620 2225 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba806585f81241b78beb15578fd9e11e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43261 } }
I20260926 20:15:58.071913 2225 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:58.071977 2225 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba806585f81241b78beb15578fd9e11e, State: Initialized, Role: FOLLOWER
I20260926 20:15:58.072499 2225 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba806585f81241b78beb15578fd9e11e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43261 } }
I20260926 20:15:58.072679 2225 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:15:58.072788 2225 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:15:58.072927 2225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:58.074059 2225 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba806585f81241b78beb15578fd9e11e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43261 } }
I20260926 20:15:58.074463 2225 leader_election.cc:302] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba806585f81241b78beb15578fd9e11e; no voters:
I20260926 20:15:58.074893 2225 leader_election.cc:288] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:15:58.074970 2230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:58.075184 2230 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [term 1 LEADER]: Becoming Leader. State: Replica: ba806585f81241b78beb15578fd9e11e, State: Running, Role: LEADER
I20260926 20:15:58.075500 2230 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba806585f81241b78beb15578fd9e11e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43261 } }
I20260926 20:15:58.076313 2225 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:58.077312 2231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba806585f81241b78beb15578fd9e11e. Latest consensus state: current_term: 1 leader_uuid: "ba806585f81241b78beb15578fd9e11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba806585f81241b78beb15578fd9e11e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43261 } } }
I20260926 20:15:58.077370 2232 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba806585f81241b78beb15578fd9e11e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba806585f81241b78beb15578fd9e11e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43261 } } }
I20260926 20:15:58.077592 2232 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:58.077620 2231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:58.078162 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:58.062783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.31.119.254:43261)
0926 20:15:58.063363 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:58.063379 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:58.063419 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:15:58.064141 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:58.064163 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:58.064601 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:58.065004 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:58.065952 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.066964 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.067958 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.068463 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.068648 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:58.071527 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:58.071533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:58.071549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:58.071578 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:58.073704 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:58.074248 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:58.074254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:58.074260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:58.074398 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:58.074759 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:58.074766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:58.074768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:58.075367 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:58.076441 (+ 1074us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:58.076612 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 20:15:58.076973 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":162,"threads_started":1}
W20260926 20:15:58.081727 2245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:15:58.081820 2245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:15:58.081938 2239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:15:58.082764 2239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:15:58.088492 2239 catalog_manager.cc:1347] Generated new cluster ID: 0e66ba1b3e8847a485b88405be603364
I20260926 20:15:58.088595 2239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:15:58.099471 2239 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:15:58.100816 2239 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:15:58.118489 2239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba806585f81241b78beb15578fd9e11e: Generated new TSK 0
I20260926 20:15:58.119441 2239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:15:58.153673 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43261
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:58.271420 2251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:58.271775 2251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:58.271870 2251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:58.276182 2251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:58.276307 2251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:58.276428 2251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:15:58.282347 2251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:58.283530 2251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:58.284616 2251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:58.294812 2259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:58.294806 2256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:58.294915 2251 server_base.cc:1034] running on GCE node
W20260926 20:15:58.294854 2257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:58.295661 2251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:58.296350 2251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:58.297545 2251 hybrid_clock.cc:648] HybridClock initialized: now 1790453758297479 us; error 77 us; skew 500 ppm
Sep 26 20:15:58 dist-test-slave-vnbh krb5kdc[2144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:58.300634 2251 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:15:58.301702 2251 webserver.cc:460] Webserver started at http://127.31.119.193:39209/ using document root <none> and password file <none>
I20260926 20:15:58.302342 2251 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:58.302443 2251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:58.302683 2251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:58.304858 2251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "7076b97862084030a5ec7349831bb28e"
format_stamp: "Formatted at 2026-09-26 20:15:58 on dist-test-slave-vnbh"
server_key: "6fe3d241f651a4b98c64764ed749c10b"
server_key_iv: "2020fc6d6319a38e22f02b5b4b30ea70"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.305541 2251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "7076b97862084030a5ec7349831bb28e"
format_stamp: "Formatted at 2026-09-26 20:15:58 on dist-test-slave-vnbh"
server_key: "6fe3d241f651a4b98c64764ed749c10b"
server_key_iv: "2020fc6d6319a38e22f02b5b4b30ea70"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.309959 2251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:15:58.312848 2266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:58.314527 2251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:15:58.314711 2251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "7076b97862084030a5ec7349831bb28e"
format_stamp: "Formatted at 2026-09-26 20:15:58 on dist-test-slave-vnbh"
server_key: "6fe3d241f651a4b98c64764ed749c10b"
server_key_iv: "2020fc6d6319a38e22f02b5b4b30ea70"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.314880 2251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:58.332226 2251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:58.335680 2251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:58.335927 2251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:58.336619 2251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:15:58.337661 2251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:58.337738 2251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:58.337819 2251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:58.337877 2251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:58.347929 2251 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:45913
I20260926 20:15:58.347949 2379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:45913 every 8 connection(s)
I20260926 20:15:58.348942 2251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:15:58.351179 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 2251
I20260926 20:15:58.351289 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:15:58.351562 32223 external_mini_cluster.cc:1428] Setting key 45c9f86bdc7b8e93a64e5c64fd63eb21
Sep 26 20:15:58 dist-test-slave-vnbh krb5kdc[2144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
I20260926 20:15:58.366757 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:58.351043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:54705 (local address 127.31.119.254:43261)
0926 20:15:58.351184 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:58.351189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:58.352396 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 20:15:58.354077 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:58.354080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:58.354134 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:58.354218 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:58.355935 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.356478 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.358471 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.358755 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.358847 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:58.362103 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:58.362110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:58.362116 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:58.362156 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:58.363843 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:58.364516 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:58.364519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:58.364520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:58.364595 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:58.365019 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:58.365023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:58.365025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:58.365361 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:58.365486 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:58.366235 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 20:15:58.366414 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:15:58.367849 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:58.351616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:43261 (local address 127.31.119.193:54705)
0926 20:15:58.352244 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:58.352288 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:58.353848 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:58.354431 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:58.354439 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:58.354988 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:58.355757 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:58.355781 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.356613 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.356619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:58.358298 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:58.358314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.358942 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.358950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:58.359223 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:58.359960 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:58.359983 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:58.361862 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:58.364002 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:58.364013 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:58.364039 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:58.364410 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:58.364688 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:58.364694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:58.364697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:58.364890 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:58.365540 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:58.365547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:58.365915 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 20:15:58.366121 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 20:15:58.366455 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":152,"threads_started":1}
I20260926 20:15:58.369467 2380 heartbeater.cc:347] Connected to a master server at 127.31.119.254:43261
I20260926 20:15:58.369794 2380 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:58.370575 2380 heartbeater.cc:511] Master 127.31.119.254:43261 requested a full tablet report, sending...
I20260926 20:15:58.372525 2189 ts_manager.cc:194] Registered new tserver with Master: 7076b97862084030a5ec7349831bb28e (127.31.119.193:45913)
I20260926 20:15:58.373828 2189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:54705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:15:58.386185 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43261
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:58.501004 2385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:58.501344 2385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:58.501390 2385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:58.505647 2385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:58.505729 2385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:58.505836 2385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:15:58.511572 2385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:58.512761 2385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:58.513880 2385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:58.524236 2393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:58.524236 2391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:58.524380 2390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:58.524570 2385 server_base.cc:1034] running on GCE node
I20260926 20:15:58.525095 2385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:58.525905 2385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:58.527076 2385 hybrid_clock.cc:648] HybridClock initialized: now 1790453758527034 us; error 55 us; skew 500 ppm
Sep 26 20:15:58 dist-test-slave-vnbh krb5kdc[2144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:58.530059 2385 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:15:58.530968 2385 webserver.cc:460] Webserver started at http://127.31.119.194:41057/ using document root <none> and password file <none>
I20260926 20:15:58.531519 2385 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:58.531577 2385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:58.531750 2385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:58.534004 2385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "17aeebe8341c4db9bee2562bc867fc65"
format_stamp: "Formatted at 2026-09-26 20:15:58 on dist-test-slave-vnbh"
server_key: "8ac66c4ba9a6713fc104754a9fc8921f"
server_key_iv: "d389d9df8478731a841ec493d2cd01a3"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.534575 2385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "17aeebe8341c4db9bee2562bc867fc65"
format_stamp: "Formatted at 2026-09-26 20:15:58 on dist-test-slave-vnbh"
server_key: "8ac66c4ba9a6713fc104754a9fc8921f"
server_key_iv: "d389d9df8478731a841ec493d2cd01a3"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.538522 2385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 20:15:58.541011 2400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:58.542372 2385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 20:15:58.542541 2385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "17aeebe8341c4db9bee2562bc867fc65"
format_stamp: "Formatted at 2026-09-26 20:15:58 on dist-test-slave-vnbh"
server_key: "8ac66c4ba9a6713fc104754a9fc8921f"
server_key_iv: "d389d9df8478731a841ec493d2cd01a3"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.542644 2385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:58.560753 2385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:58.565975 2385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:58.566223 2385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:58.567067 2385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:15:58.568277 2385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:58.568354 2385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:58.568424 2385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:58.568459 2385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:58.579300 2385 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:34283
I20260926 20:15:58.579319 2513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:34283 every 8 connection(s)
I20260926 20:15:58.580344 2385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:15:58.583323 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 2385
I20260926 20:15:58.583432 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:15:58.583704 32223 external_mini_cluster.cc:1428] Setting key a0ec4661838c5b15eb2e5f60b5e2b835
Sep 26 20:15:58 dist-test-slave-vnbh krb5kdc[2144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
I20260926 20:15:58.598116 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:58.582445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:52273 (local address 127.31.119.254:43261)
0926 20:15:58.582578 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:58.582587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:58.583520 (+ 933us) server_negotiation.cc:407] Connection header received
0926 20:15:58.585154 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:58.585158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:58.585210 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:58.585329 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:58.587462 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.588430 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.590529 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.590821 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.590993 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:58.593763 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:58.593768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:58.593771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:58.593811 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:58.595451 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:58.596178 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:58.596181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:58.596183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:58.596265 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:58.596692 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:58.596695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:58.596696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:58.596903 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:58.596998 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:58.597760 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 20:15:58.597933 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:15:58.599112 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:58.582845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:43261 (local address 127.31.119.194:52273)
0926 20:15:58.583339 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:58.583410 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:58.584901 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:58.585843 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:58.585855 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:58.586445 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:58.587244 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:58.587272 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.588650 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.588657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:58.590331 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:58.590349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.591028 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.591032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:58.591241 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:58.591932 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:58.591951 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:58.593539 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:58.595576 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:58.595582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:58.595599 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:58.596040 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:58.596356 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:58.596358 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:58.596359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:58.596517 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:58.597019 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:58.597025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:58.597392 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 20:15:58.597680 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 20:15:58.598060 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":153,"threads_started":1}
I20260926 20:15:58.600551 2514 heartbeater.cc:347] Connected to a master server at 127.31.119.254:43261
I20260926 20:15:58.600824 2514 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:58.601447 2514 heartbeater.cc:511] Master 127.31.119.254:43261 requested a full tablet report, sending...
I20260926 20:15:58.602768 2189 ts_manager.cc:194] Registered new tserver with Master: 17aeebe8341c4db9bee2562bc867fc65 (127.31.119.194:34283)
I20260926 20:15:58.603454 2189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:52273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:15:58.617377 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43261
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:58.735270 2519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:58.735543 2519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:58.735586 2519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:58.739606 2519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:58.739679 2519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:58.739740 2519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:15:58.745313 2519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:58.746496 2519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:58.747561 2519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:58.757997 2524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:58.757975 2519 server_base.cc:1034] running on GCE node
W20260926 20:15:58.758061 2527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:58.757978 2525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:58.758731 2519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:58.759426 2519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:58.760744 2519 hybrid_clock.cc:648] HybridClock initialized: now 1790453758760708 us; error 56 us; skew 500 ppm
Sep 26 20:15:58 dist-test-slave-vnbh krb5kdc[2144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:58.763904 2519 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:15:58.764976 2519 webserver.cc:460] Webserver started at http://127.31.119.195:33669/ using document root <none> and password file <none>
I20260926 20:15:58.765672 2519 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:58.765744 2519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:58.765965 2519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:58.768090 2519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "5926e8e07d874e79a868f5fa342f3883"
format_stamp: "Formatted at 2026-09-26 20:15:58 on dist-test-slave-vnbh"
server_key: "0e53c42ad9aadff836b646138964f4d1"
server_key_iv: "aaa2118db31c382b53c1b4f17d969a9a"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.768679 2519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "5926e8e07d874e79a868f5fa342f3883"
format_stamp: "Formatted at 2026-09-26 20:15:58 on dist-test-slave-vnbh"
server_key: "0e53c42ad9aadff836b646138964f4d1"
server_key_iv: "aaa2118db31c382b53c1b4f17d969a9a"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.772675 2519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 20:15:58.775439 2534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:58.776978 2519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:58.777124 2519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "5926e8e07d874e79a868f5fa342f3883"
format_stamp: "Formatted at 2026-09-26 20:15:58 on dist-test-slave-vnbh"
server_key: "0e53c42ad9aadff836b646138964f4d1"
server_key_iv: "aaa2118db31c382b53c1b4f17d969a9a"
server_key_version: "encryptionkey@0"
I20260926 20:15:58.777249 2519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:58.789376 2519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:58.792516 2519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:58.792738 2519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:58.793448 2519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:15:58.794320 2519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:15:58.794368 2519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:58.794433 2519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:15:58.794471 2519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:58.804181 2519 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:45383
I20260926 20:15:58.804203 2647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:45383 every 8 connection(s)
I20260926 20:15:58.805181 2519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:15:58.814832 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 2519
I20260926 20:15:58.814992 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
Sep 26 20:15:58 dist-test-slave-vnbh krb5kdc[2144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:15:58.815346 32223 external_mini_cluster.cc:1428] Setting key 2479ee00f380f5d21c9c6c39a34edefb
I20260926 20:15:58.820004 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:58.807240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:44239 (local address 127.31.119.254:43261)
0926 20:15:58.807395 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:58.807400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:58.808355 (+ 955us) server_negotiation.cc:407] Connection header received
0926 20:15:58.809819 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:58.809822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:58.809872 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:58.809950 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:58.811189 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.811719 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.812893 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.813110 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.813216 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:58.816039 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:58.816043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:58.816046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:58.816080 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:58.817919 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:58.818489 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:58.818492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:58.818493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:58.818579 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:58.818908 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:58.818910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:58.818911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:58.819149 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:58.819231 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:58.819774 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 20:15:58.819880 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:15:58.820969 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:58.807704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:43261 (local address 127.31.119.195:44239)
0926 20:15:58.808194 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:58.808240 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:58.809601 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:58.810116 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:58.810123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:58.810522 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:58.811029 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:58.811044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.811831 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.811833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:58.812768 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:58.812777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:58.813275 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:58.813327 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:58.813492 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:58.814080 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:58.814097 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:58.815841 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:58.818075 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:58.818082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:58.818101 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:58.818388 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:58.818671 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:58.818675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:58.818677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:58.818822 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:58.819269 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:58.819275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:58.819549 (+ 274us) client_negotiation.cc:769] Sending connection context
0926 20:15:58.819723 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 20:15:58.820001 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":116,"threads_started":1}
I20260926 20:15:58.822202 2648 heartbeater.cc:347] Connected to a master server at 127.31.119.254:43261
I20260926 20:15:58.822463 2648 heartbeater.cc:464] Registering TS with master...
I20260926 20:15:58.822969 2648 heartbeater.cc:511] Master 127.31.119.254:43261 requested a full tablet report, sending...
I20260926 20:15:58.824007 2189 ts_manager.cc:194] Registered new tserver with Master: 5926e8e07d874e79a868f5fa342f3883 (127.31.119.195:45383)
I20260926 20:15:58.824579 2189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:44239
I20260926 20:15:58.829857 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:15:58.843546 2228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:58.841834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.31.119.254:43261)
0926 20:15:58.841998 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:58.842004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:58.842199 (+ 195us) server_negotiation.cc:407] Connection header received
0926 20:15:58.842367 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:58.842370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:58.842424 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:58.842515 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:58.843376 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47850: BlockingRecv error: recv got EOF from 127.0.0.1:47850 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:15:58.844877 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 2251
I20260926 20:15:58.850644 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 2385
I20260926 20:15:58.856642 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 2519
I20260926 20:15:58.862419 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 2157
2026-09-26T20:15:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2191 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:15:58 dist-test-slave-vnbh krb5kdc[2660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:15:58 dist-test-slave-vnbh krb5kdc[2660](info): set up 2 sockets
Sep 26 20:15:58 dist-test-slave-vnbh krb5kdc[2660](info): commencing operation
krb5kdc: starting...
W20260926 20:15:59.930261 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:00 dist-test-slave-vnbh krb5kdc[2660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:16:00.062469 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:42869
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:00.178275 2673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:00.178589 2673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:00.178633 2673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:00.182969 2673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:00.183041 2673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:00.183061 2673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:00.183081 2673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:00.183097 2673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:00.188856 2673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:42869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:00.190130 2673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:00.191305 2673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:00.203482 2679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:00.203554 2678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:00.203544 2673 server_base.cc:1034] running on GCE node
W20260926 20:16:00.203826 2681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:00.204509 2673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:00.205984 2673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:00.207203 2673 hybrid_clock.cc:648] HybridClock initialized: now 1790453760207189 us; error 33 us; skew 500 ppm
Sep 26 20:16:00 dist-test-slave-vnbh krb5kdc[2660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453760, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:00.216049 2673 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:16:00.217128 2673 webserver.cc:460] Webserver started at http://127.31.119.254:37813/ using document root <none> and password file <none>
I20260926 20:16:00.217801 2673 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:00.217870 2673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:00.218053 2673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:00.220254 2673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "04c6d51653444c19a3c6ffb1b5a82062"
format_stamp: "Formatted at 2026-09-26 20:16:00 on dist-test-slave-vnbh"
server_key: "55a05470ca07df7b8b893c0880ca1ffa"
server_key_iv: "04e02a712c97483b8f4a87706e4d97c3"
server_key_version: "encryptionkey@0"
I20260926 20:16:00.220813 2673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "04c6d51653444c19a3c6ffb1b5a82062"
format_stamp: "Formatted at 2026-09-26 20:16:00 on dist-test-slave-vnbh"
server_key: "55a05470ca07df7b8b893c0880ca1ffa"
server_key_iv: "04e02a712c97483b8f4a87706e4d97c3"
server_key_version: "encryptionkey@0"
I20260926 20:16:00.224885 2673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:16:00.227617 2688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:00.229161 2673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:16:00.229349 2673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "04c6d51653444c19a3c6ffb1b5a82062"
format_stamp: "Formatted at 2026-09-26 20:16:00 on dist-test-slave-vnbh"
server_key: "55a05470ca07df7b8b893c0880ca1ffa"
server_key_iv: "04e02a712c97483b8f4a87706e4d97c3"
server_key_version: "encryptionkey@0"
I20260926 20:16:00.229489 2673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:00.243323 2673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:00.246876 2673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:00.247143 2673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:00.254828 2673 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:42869
I20260926 20:16:00.255126 2740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:42869 every 8 connection(s)
I20260926 20:16:00.255955 2673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:00.259176 2741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:00.259779 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 2673
I20260926 20:16:00.259892 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:00.260201 32223 external_mini_cluster.cc:1428] Setting key 7f8a7e5ae02df551a1a31622aae035d0
I20260926 20:16:00.266924 2741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062: Bootstrap starting.
I20260926 20:16:00.269726 2741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:00.270542 2741 log.cc:824] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:16:00 dist-test-slave-vnbh krb5kdc[2660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:00.273052 2741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062: No bootstrap required, opened a new log
I20260926 20:16:00.277500 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:00.261952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.31.119.254:42869)
0926 20:16:00.262470 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:00.262491 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:00.262539 (+ 48us) server_negotiation.cc:407] Connection header received
0926 20:16:00.263433 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:00.263459 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:00.263938 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:00.264378 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:00.265222 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:00.266816 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:00.267887 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:00.268624 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:00.268795 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:00.271522 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:00.271529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:00.271547 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:00.271587 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:00.273825 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:00.274329 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:00.274335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:00.274342 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:00.274487 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:00.274853 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:00.274856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:00.274858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:00.275473 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:00.275724 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:00.276044 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 20:16:00.276374 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":137,"threads_started":1}
I20260926 20:16:00.278038 2741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c6d51653444c19a3c6ffb1b5a82062" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42869 } }
I20260926 20:16:00.278339 2741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:00.278414 2741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c6d51653444c19a3c6ffb1b5a82062, State: Initialized, Role: FOLLOWER
I20260926 20:16:00.278887 2741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c6d51653444c19a3c6ffb1b5a82062" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42869 } }
I20260926 20:16:00.279013 2741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:00.279110 2741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:00.279213 2741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:00.280490 2741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c6d51653444c19a3c6ffb1b5a82062" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42869 } }
I20260926 20:16:00.280905 2741 leader_election.cc:302] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04c6d51653444c19a3c6ffb1b5a82062; no voters:
I20260926 20:16:00.280965 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:16:00.281255 2741 leader_election.cc:288] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:00.281389 2746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:00.281615 2746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [term 1 LEADER]: Becoming Leader. State: Replica: 04c6d51653444c19a3c6ffb1b5a82062, State: Running, Role: LEADER
I20260926 20:16:00.281977 2746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c6d51653444c19a3c6ffb1b5a82062" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42869 } }
I20260926 20:16:00.282969 2741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:00.283551 2746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c6d51653444c19a3c6ffb1b5a82062. Latest consensus state: current_term: 1 leader_uuid: "04c6d51653444c19a3c6ffb1b5a82062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c6d51653444c19a3c6ffb1b5a82062" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42869 } } }
I20260926 20:16:00.283685 2746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:00.283981 2747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04c6d51653444c19a3c6ffb1b5a82062" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c6d51653444c19a3c6ffb1b5a82062" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42869 } } }
I20260926 20:16:00.284087 2747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062 [sys.catalog]: This master's current role is: LEADER
W20260926 20:16:00.288199 2762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:16:00.288292 2762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:16:00.289233 2752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:00.290349 2752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:00.296525 2752 catalog_manager.cc:1347] Generated new cluster ID: f032f2d162d041b5a32b7beafb8be8b3
I20260926 20:16:00.296605 2752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:00.306763 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:00.297062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.31.119.254:42869)
0926 20:16:00.297221 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:00.297226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:00.301753 (+ 4527us) server_negotiation.cc:407] Connection header received
0926 20:16:00.301946 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:00.301949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:00.302005 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:00.302093 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:00.302687 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:00.303251 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:00.304121 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:00.304389 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:00.304499 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:00.305508 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:00.305512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:00.305515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:00.306216 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:00.306257 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:00.306278 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:00.306378 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:00.306542 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:16:00.306612 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":49}
I20260926 20:16:00.307891 2752 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:00.308842 2752 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:00.318624 2752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04c6d51653444c19a3c6ffb1b5a82062: Generated new TSK 0
I20260926 20:16:00.319585 2752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:16:00.331887 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 2673
2026-09-26T20:16:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1466 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:16:00 dist-test-slave-vnbh krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:16:00 dist-test-slave-vnbh krb5kdc[2774](info): set up 2 sockets
Sep 26 20:16:00 dist-test-slave-vnbh krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260926 20:16:01.398425 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:01 dist-test-slave-vnbh krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:16:01.538859 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:43995
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:43995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:01.659893 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:01.660250 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:01.660336 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:01.665647 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:01.665752 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:01.665791 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:01.665853 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:01.665916 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:01.672631 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:43995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:43995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:01.674054 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:01.675278 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:01.685510 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:01.685547 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:01.685801 2787 server_base.cc:1034] running on GCE node
W20260926 20:16:01.685751 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:01.686524 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:01.687871 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:01.689271 2787 hybrid_clock.cc:648] HybridClock initialized: now 1790453761689244 us; error 46 us; skew 500 ppm
Sep 26 20:16:01 dist-test-slave-vnbh krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453761, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:01.692759 2787 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:16:01.694129 2787 webserver.cc:460] Webserver started at http://127.31.119.254:36357/ using document root <none> and password file <none>
I20260926 20:16:01.694836 2787 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:01.694950 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:01.695226 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:01.697762 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "1234ef4dc8e54d14a03adc755bff7146"
format_stamp: "Formatted at 2026-09-26 20:16:01 on dist-test-slave-vnbh"
server_key: "cc3236720ac5e3285d2b7286ea908b05"
server_key_iv: "f642285fd3ef69b9d3e8a0440b4c5fc4"
server_key_version: "encryptionkey@0"
I20260926 20:16:01.698427 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "1234ef4dc8e54d14a03adc755bff7146"
format_stamp: "Formatted at 2026-09-26 20:16:01 on dist-test-slave-vnbh"
server_key: "cc3236720ac5e3285d2b7286ea908b05"
server_key_iv: "f642285fd3ef69b9d3e8a0440b4c5fc4"
server_key_version: "encryptionkey@0"
I20260926 20:16:01.702936 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:01.706210 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:01.707947 2787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:16:01.708148 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "1234ef4dc8e54d14a03adc755bff7146"
format_stamp: "Formatted at 2026-09-26 20:16:01 on dist-test-slave-vnbh"
server_key: "cc3236720ac5e3285d2b7286ea908b05"
server_key_iv: "f642285fd3ef69b9d3e8a0440b4c5fc4"
server_key_version: "encryptionkey@0"
I20260926 20:16:01.708303 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:01.729038 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:01.733131 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:01.733446 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:01.742810 2787 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:43995
I20260926 20:16:01.743134 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:43995 every 8 connection(s)
I20260926 20:16:01.743963 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:01.747011 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 2787
I20260926 20:16:01.747197 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:01.747377 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:01.747562 32223 external_mini_cluster.cc:1428] Setting key e6181c5820efc902770158acc0baa12f
I20260926 20:16:01.755278 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146: Bootstrap starting.
Sep 26 20:16:01 dist-test-slave-vnbh krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:01.758126 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:01.759047 2855 log.cc:824] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:01.761861 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146: No bootstrap required, opened a new log
I20260926 20:16:01.765064 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:01.749228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.31.119.254:43995)
0926 20:16:01.749806 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:01.749825 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:01.749873 (+ 48us) server_negotiation.cc:407] Connection header received
0926 20:16:01.750591 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:01.750616 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:01.751118 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:01.751536 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:01.752367 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:01.753957 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:01.754894 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:01.755577 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:01.755755 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:01.758669 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:01.758676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:01.758694 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:01.758736 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:01.761084 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:01.761665 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:01.761671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:01.761677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:01.761825 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:01.762221 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:01.762224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:01.762226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:01.762888 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:01.763106 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:01.763461 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 20:16:01.763811 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":153,"threads_started":1}
I20260926 20:16:01.766870 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1234ef4dc8e54d14a03adc755bff7146" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43995 } }
I20260926 20:16:01.767204 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:01.767295 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1234ef4dc8e54d14a03adc755bff7146, State: Initialized, Role: FOLLOWER
I20260926 20:16:01.767867 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1234ef4dc8e54d14a03adc755bff7146" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43995 } }
I20260926 20:16:01.768071 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:01.768157 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:01.768267 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:01.769444 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1234ef4dc8e54d14a03adc755bff7146" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43995 } }
I20260926 20:16:01.769845 2855 leader_election.cc:302] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1234ef4dc8e54d14a03adc755bff7146; no voters:
I20260926 20:16:01.770242 2855 leader_election.cc:288] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:01.770407 2860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:01.770689 2860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [term 1 LEADER]: Becoming Leader. State: Replica: 1234ef4dc8e54d14a03adc755bff7146, State: Running, Role: LEADER
I20260926 20:16:01.771068 2860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1234ef4dc8e54d14a03adc755bff7146" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43995 } }
I20260926 20:16:01.771744 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:01.772807 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1234ef4dc8e54d14a03adc755bff7146. Latest consensus state: current_term: 1 leader_uuid: "1234ef4dc8e54d14a03adc755bff7146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1234ef4dc8e54d14a03adc755bff7146" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43995 } } }
I20260926 20:16:01.772950 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:01.773157 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1234ef4dc8e54d14a03adc755bff7146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1234ef4dc8e54d14a03adc755bff7146" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43995 } } }
I20260926 20:16:01.773383 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:01.773646 2869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:01.777208 2869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:01.783890 2869 catalog_manager.cc:1347] Generated new cluster ID: 5b66e7b12cb340adacaaacb6f2b87317
I20260926 20:16:01.783985 2869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:01.799367 2869 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:01.800755 2869 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:01.808624 2869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1234ef4dc8e54d14a03adc755bff7146: Generated new TSK 0
I20260926 20:16:01.809653 2869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:16:01.854115 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43995
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:01.981716 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:01.982018 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:01.982064 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:01.986493 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:01.986580 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:01.986649 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:01.992332 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:01.993530 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:01.994623 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:02.005625 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:02.005635 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:02.005785 2881 server_base.cc:1034] running on GCE node
W20260926 20:16:02.005985 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:02.006601 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:02.007505 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:02.008754 2881 hybrid_clock.cc:648] HybridClock initialized: now 1790453762008733 us; error 40 us; skew 500 ppm
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:02.012058 2881 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:16:02.013269 2881 webserver.cc:460] Webserver started at http://127.31.119.193:34335/ using document root <none> and password file <none>
I20260926 20:16:02.014019 2881 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:02.014087 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:02.014268 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:02.016453 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "bb98e59c1e5e434fbfc5dcbde4d07108"
format_stamp: "Formatted at 2026-09-26 20:16:02 on dist-test-slave-vnbh"
server_key: "4cdfdfac6991f3a58cde7410031f524d"
server_key_iv: "00b8c5fb238910a19aa845b5466c14de"
server_key_version: "encryptionkey@0"
I20260926 20:16:02.017020 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "bb98e59c1e5e434fbfc5dcbde4d07108"
format_stamp: "Formatted at 2026-09-26 20:16:02 on dist-test-slave-vnbh"
server_key: "4cdfdfac6991f3a58cde7410031f524d"
server_key_iv: "00b8c5fb238910a19aa845b5466c14de"
server_key_version: "encryptionkey@0"
I20260926 20:16:02.021338 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 20:16:02.024392 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:02.025998 2881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:02.026149 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "bb98e59c1e5e434fbfc5dcbde4d07108"
format_stamp: "Formatted at 2026-09-26 20:16:02 on dist-test-slave-vnbh"
server_key: "4cdfdfac6991f3a58cde7410031f524d"
server_key_iv: "00b8c5fb238910a19aa845b5466c14de"
server_key_version: "encryptionkey@0"
I20260926 20:16:02.026268 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:02.052347 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:02.056003 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:02.056178 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:02.056757 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:02.057677 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:02.057729 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:02.057770 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:02.057786 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:02.067381 2881 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:37995
I20260926 20:16:02.067416 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:37995 every 8 connection(s)
I20260926 20:16:02.068769 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:02.072568 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 2881
I20260926 20:16:02.072723 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:16:02.073082 32223 external_mini_cluster.cc:1428] Setting key 66f5f58643bbd98fa6f45e3a29357867
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
I20260926 20:16:02.088923 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:02.071352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:38335 (local address 127.31.119.254:43995)
0926 20:16:02.071533 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:02.071539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:02.072876 (+ 1337us) server_negotiation.cc:407] Connection header received
0926 20:16:02.074696 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:02.074700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:02.074758 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:02.074834 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:02.078208 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.078933 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.080389 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.080646 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.080773 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:02.083782 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:02.083785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:02.083788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:02.083817 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:02.085783 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:02.086610 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:02.086614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:02.086615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:02.086719 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:02.087242 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:02.087245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:02.087246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:02.087482 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:02.087651 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:02.088506 (+ 855us) server_negotiation.cc:299] Negotiation successful
0926 20:16:02.088691 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 20:16:02.089601 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:02.071832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:43995 (local address 127.31.119.193:38335)
0926 20:16:02.072603 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:02.072722 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:02.074436 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:02.076656 (+ 2220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:02.076667 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:02.077104 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:02.077963 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:02.077992 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.079140 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.079143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:02.080216 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:02.080229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.080848 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.080853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:02.081052 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:02.081881 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:02.081900 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:02.083555 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:02.085956 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:02.085963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:02.085980 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:02.086431 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:02.086828 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:02.086835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:02.086840 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:02.087075 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:02.087600 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:02.087612 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:02.088059 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 20:16:02.088281 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 20:16:02.088590 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":311,"threads_started":1}
I20260926 20:16:02.091151 3010 heartbeater.cc:347] Connected to a master server at 127.31.119.254:43995
I20260926 20:16:02.091468 3010 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:02.092200 3010 heartbeater.cc:511] Master 127.31.119.254:43995 requested a full tablet report, sending...
I20260926 20:16:02.094184 2819 ts_manager.cc:194] Registered new tserver with Master: bb98e59c1e5e434fbfc5dcbde4d07108 (127.31.119.193:37995)
I20260926 20:16:02.095422 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:38335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:16:02.107029 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43995
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:02.227170 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:02.227473 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:02.227559 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:02.231884 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:02.231964 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:02.232028 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:02.237808 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:02.239030 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:02.240155 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:02.251147 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:02.251178 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:02.251151 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:02.251245 3015 server_base.cc:1034] running on GCE node
I20260926 20:16:02.251991 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:02.252801 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:02.254034 3015 hybrid_clock.cc:648] HybridClock initialized: now 1790453762253941 us; error 60 us; skew 500 ppm
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:02.257381 3015 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:16:02.258481 3015 webserver.cc:460] Webserver started at http://127.31.119.194:42927/ using document root <none> and password file <none>
I20260926 20:16:02.259123 3015 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:02.259233 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:02.259488 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:02.261925 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "e16c0fcc842e44d2835008d347b5f30e"
format_stamp: "Formatted at 2026-09-26 20:16:02 on dist-test-slave-vnbh"
server_key: "52e466909b12ebe930557e87d34cc15f"
server_key_iv: "b4f7bcf3eb879fb7ca16d9331539acd3"
server_key_version: "encryptionkey@0"
I20260926 20:16:02.262612 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "e16c0fcc842e44d2835008d347b5f30e"
format_stamp: "Formatted at 2026-09-26 20:16:02 on dist-test-slave-vnbh"
server_key: "52e466909b12ebe930557e87d34cc15f"
server_key_iv: "b4f7bcf3eb879fb7ca16d9331539acd3"
server_key_version: "encryptionkey@0"
I20260926 20:16:02.267381 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 20:16:02.270555 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:02.272037 3015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:16:02.272233 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "e16c0fcc842e44d2835008d347b5f30e"
format_stamp: "Formatted at 2026-09-26 20:16:02 on dist-test-slave-vnbh"
server_key: "52e466909b12ebe930557e87d34cc15f"
server_key_iv: "b4f7bcf3eb879fb7ca16d9331539acd3"
server_key_version: "encryptionkey@0"
I20260926 20:16:02.272368 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:02.296007 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:02.299714 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:02.299937 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:02.300583 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:02.301558 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:02.301625 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:02.301694 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:02.301739 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:02.311832 3015 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:34265
I20260926 20:16:02.311849 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:34265 every 8 connection(s)
I20260926 20:16:02.312842 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:02.314750 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 3015
I20260926 20:16:02.314893 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:02.315259 32223 external_mini_cluster.cc:1428] Setting key 78ce4cbab138c1c31a7f54adf966eb75
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
I20260926 20:16:02.330983 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:02.314898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:59427 (local address 127.31.119.254:43995)
0926 20:16:02.315050 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:02.315055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:02.316237 (+ 1182us) server_negotiation.cc:407] Connection header received
0926 20:16:02.318230 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:02.318234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:02.318286 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:02.318371 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:02.319877 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.320826 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.322315 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.322788 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.322942 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:02.326103 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:02.326106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:02.326108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:02.326139 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:02.328196 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:02.329004 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:02.329011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:02.329016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:02.329117 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:02.329623 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:02.329626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:02.329628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:02.329839 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:02.330025 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:02.330647 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 20:16:02.330805 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:16:02.331995 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:02.315432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:43995 (local address 127.31.119.194:59427)
0926 20:16:02.316076 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:02.316140 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:02.317967 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:02.318672 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:02.318685 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:02.319171 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:02.319703 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:02.319718 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.321006 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.321011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:02.322161 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:02.322173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.322988 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.322991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:02.323184 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:02.323889 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:02.323907 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:02.325870 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:02.328387 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:02.328394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:02.328408 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:02.328874 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:02.329260 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:02.329263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:02.329266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:02.329494 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:02.330020 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:02.330028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:02.330385 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 20:16:02.330594 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 20:16:02.330987 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":173,"threads_started":1}
I20260926 20:16:02.333482 3144 heartbeater.cc:347] Connected to a master server at 127.31.119.254:43995
I20260926 20:16:02.333775 3144 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:02.334334 3144 heartbeater.cc:511] Master 127.31.119.254:43995 requested a full tablet report, sending...
I20260926 20:16:02.335487 2819 ts_manager.cc:194] Registered new tserver with Master: e16c0fcc842e44d2835008d347b5f30e (127.31.119.194:34265)
I20260926 20:16:02.336093 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:59427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:16:02.350145 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43995
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:02.469596 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:02.469915 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:02.469964 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:02.474442 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:02.474520 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:02.474586 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:02.480587 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:43995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:02.481827 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:02.482908 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:02.493475 3149 server_base.cc:1034] running on GCE node
W20260926 20:16:02.493417 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:02.493423 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:02.493423 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:02.494058 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:02.494808 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:02.495985 3149 hybrid_clock.cc:648] HybridClock initialized: now 1790453762495941 us; error 59 us; skew 500 ppm
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:02.499034 3149 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:16:02.500053 3149 webserver.cc:460] Webserver started at http://127.31.119.195:42241/ using document root <none> and password file <none>
I20260926 20:16:02.500656 3149 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:02.500749 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:02.500990 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:02.503263 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "a91270d39d5c49aca640e598c7ad09d7"
format_stamp: "Formatted at 2026-09-26 20:16:02 on dist-test-slave-vnbh"
server_key: "7237187fd9ae71667470bbe9b6da1da2"
server_key_iv: "1254c72812d579e3d187834c09b989a1"
server_key_version: "encryptionkey@0"
I20260926 20:16:02.503878 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "a91270d39d5c49aca640e598c7ad09d7"
format_stamp: "Formatted at 2026-09-26 20:16:02 on dist-test-slave-vnbh"
server_key: "7237187fd9ae71667470bbe9b6da1da2"
server_key_iv: "1254c72812d579e3d187834c09b989a1"
server_key_version: "encryptionkey@0"
I20260926 20:16:02.508045 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:16:02.510814 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:02.512235 3149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:16:02.512391 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "a91270d39d5c49aca640e598c7ad09d7"
format_stamp: "Formatted at 2026-09-26 20:16:02 on dist-test-slave-vnbh"
server_key: "7237187fd9ae71667470bbe9b6da1da2"
server_key_iv: "1254c72812d579e3d187834c09b989a1"
server_key_version: "encryptionkey@0"
I20260926 20:16:02.512516 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:02.530879 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:02.534588 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:02.534816 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:02.535463 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:02.536346 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:02.536417 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:02.536492 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:02.536535 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:02.546437 3149 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:40963
I20260926 20:16:02.546489 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:40963 every 8 connection(s)
I20260926 20:16:02.547472 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:02.557722 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 3149
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:02.557837 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:02.558255 32223 external_mini_cluster.cc:1428] Setting key 581d3255f3845b4c5e5a91c39cf03788
I20260926 20:16:02.563030 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:02.549500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:39761 (local address 127.31.119.254:43995)
0926 20:16:02.549756 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:02.549760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:02.550680 (+ 920us) server_negotiation.cc:407] Connection header received
0926 20:16:02.552165 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:02.552169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:02.552230 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:02.552341 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:02.553652 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.554196 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.555407 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.555645 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.555764 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:02.558666 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:02.558670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:02.558672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:02.558700 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:02.560549 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:02.561343 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:02.561347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:02.561349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:02.561441 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:02.561815 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:02.561818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:02.561820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:02.562032 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:02.562139 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:02.562718 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 20:16:02.562853 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260926 20:16:02.563948 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:02.549981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:43995 (local address 127.31.119.195:39761)
0926 20:16:02.550466 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:02.550513 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:02.551913 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:02.552529 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:02.552537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:02.552934 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:02.553490 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:02.553504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.554335 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.554338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:02.555267 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:02.555276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.555815 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.555819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:02.556012 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:02.556657 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:02.556677 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:02.558448 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:02.560704 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:02.560710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:02.560726 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:02.561199 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:02.561538 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:02.561542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:02.561544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:02.561709 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:02.562138 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:02.562147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:02.562434 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 20:16:02.562664 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 20:16:02.563036 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
I20260926 20:16:02.565447 3278 heartbeater.cc:347] Connected to a master server at 127.31.119.254:43995
I20260926 20:16:02.565799 3278 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:02.566469 3278 heartbeater.cc:511] Master 127.31.119.254:43995 requested a full tablet report, sending...
I20260926 20:16:02.567977 2819 ts_manager.cc:194] Registered new tserver with Master: a91270d39d5c49aca640e598c7ad09d7 (127.31.119.195:40963)
I20260926 20:16:02.568837 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:39761
I20260926 20:16:02.573369 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453762, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453762, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:02.602135 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:02.592373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.31.119.254:43995)
0926 20:16:02.592565 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:02.592569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:02.592719 (+ 150us) server_negotiation.cc:407] Connection header received
0926 20:16:02.592913 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:02.592916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:02.592963 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:02.593130 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:02.593971 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.594553 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.595467 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:02.595698 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:02.595813 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:02.598671 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:02.598674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:02.598677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:02.598711 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:02.600141 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:02.600629 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:02.600633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:02.600635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:02.600731 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:02.601037 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:02.601040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:02.601042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:02.601406 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:02.601561 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:02.601825 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 20:16:02.601950 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 20:16:02.602741 2819 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47500
I20260926 20:16:02.603885 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 2881
I20260926 20:16:02.609838 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 3015
I20260926 20:16:02.615588 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 3149
I20260926 20:16:02.621693 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 2787
2026-09-26T20:16:02Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2293 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[3290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[3290](info): set up 2 sockets
Sep 26 20:16:02 dist-test-slave-vnbh krb5kdc[3290](info): commencing operation
krb5kdc: starting...
W20260926 20:16:03.682471 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:03 dist-test-slave-vnbh krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:16:03.813448 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:40849
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:40849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:03.929919 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:03.930207 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:03.930248 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:03.934353 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:03.934425 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:03.934446 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:03.934463 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:03.934523 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:03.940181 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:40849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:03.941391 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:03.942559 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:03.952520 3303 server_base.cc:1034] running on GCE node
W20260926 20:16:03.952442 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:03.952543 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:03.952800 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:03.953482 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:03.954746 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:03.955958 3303 hybrid_clock.cc:648] HybridClock initialized: now 1790453763955936 us; error 35 us; skew 500 ppm
Sep 26 20:16:03 dist-test-slave-vnbh krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453763, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:03.959287 3303 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:16:03.960369 3303 webserver.cc:460] Webserver started at http://127.31.119.254:37995/ using document root <none> and password file <none>
I20260926 20:16:03.960984 3303 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:03.961078 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:03.961383 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:03.963642 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2"
format_stamp: "Formatted at 2026-09-26 20:16:03 on dist-test-slave-vnbh"
server_key: "0d41e4252124f095cba41bdfa31e050f"
server_key_iv: "62b9366e57027c2d3b902891d64d2590"
server_key_version: "encryptionkey@0"
I20260926 20:16:03.964257 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2"
format_stamp: "Formatted at 2026-09-26 20:16:03 on dist-test-slave-vnbh"
server_key: "0d41e4252124f095cba41bdfa31e050f"
server_key_iv: "62b9366e57027c2d3b902891d64d2590"
server_key_version: "encryptionkey@0"
I20260926 20:16:03.968454 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:16:03.971334 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:03.972831 3303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:16:03.973044 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2"
format_stamp: "Formatted at 2026-09-26 20:16:03 on dist-test-slave-vnbh"
server_key: "0d41e4252124f095cba41bdfa31e050f"
server_key_iv: "62b9366e57027c2d3b902891d64d2590"
server_key_version: "encryptionkey@0"
I20260926 20:16:03.973196 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:03.991698 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:03.996467 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:03.996718 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:04.004413 3303 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:40849
I20260926 20:16:04.004429 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:40849 every 8 connection(s)
I20260926 20:16:04.005584 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:04.008711 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:04.010733 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 3303
I20260926 20:16:04.010877 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:04.011188 32223 external_mini_cluster.cc:1428] Setting key 276bce0f0b0edabfe18e31f589342f25
I20260926 20:16:04.015866 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2: Bootstrap starting.
I20260926 20:16:04.018460 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:04.019452 3371 log.cc:824] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:04.021832 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2: No bootstrap required, opened a new log
I20260926 20:16:04.025519 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40849 } }
I20260926 20:16:04.025861 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:04.025947 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039586b1bb3c4681ab1c33bf7ac5a7d2, State: Initialized, Role: FOLLOWER
I20260926 20:16:04.026463 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40849 } }
I20260926 20:16:04.026597 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:04.026684 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:04.026784 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:04.026934 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:04.012874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.31.119.254:40849)
0926 20:16:04.013425 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:04.013448 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:04.013487 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:16:04.014179 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:04.014199 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:04.014581 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:04.014972 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:04.015861 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.016944 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.017952 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.018469 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.018667 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:04.021599 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:04.021604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:04.021620 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:04.021657 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:04.023556 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.024071 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.024084 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.024090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.024216 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.024523 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.024527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.024529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.024976 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:04.025207 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:04.025532 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 20:16:04.025921 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":160,"threads_started":1}
I20260926 20:16:04.027823 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40849 } }
I20260926 20:16:04.028223 3371 leader_election.cc:302] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 039586b1bb3c4681ab1c33bf7ac5a7d2; no voters:
I20260926 20:16:04.028579 3371 leader_election.cc:288] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:04.028697 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:04.028939 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [term 1 LEADER]: Becoming Leader. State: Replica: 039586b1bb3c4681ab1c33bf7ac5a7d2, State: Running, Role: LEADER
I20260926 20:16:04.029887 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:04.029896 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40849 } }
I20260926 20:16:04.031749 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40849 } } }
I20260926 20:16:04.031903 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:04.032251 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 039586b1bb3c4681ab1c33bf7ac5a7d2. Latest consensus state: current_term: 1 leader_uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039586b1bb3c4681ab1c33bf7ac5a7d2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40849 } } }
I20260926 20:16:04.033164 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:04.032836 3390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:04.036059 3390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:04.042335 3390 catalog_manager.cc:1347] Generated new cluster ID: c4b7ec7553e34f96b3b0e2fae33706d1
I20260926 20:16:04.042429 3390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:04.056919 3390 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:04.058298 3390 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:04.073683 3390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 039586b1bb3c4681ab1c33bf7ac5a7d2: Generated new TSK 0
I20260926 20:16:04.074671 3390 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:16:04.118883 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40849
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:04.241751 3397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:04.242101 3397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:04.242177 3397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:04.246641 3397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:04.246749 3397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:04.246846 3397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:04.252934 3397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:04.254220 3397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:04.255335 3397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:04.267360 3397 server_base.cc:1034] running on GCE node
W20260926 20:16:04.267264 3405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:04.267256 3402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:04.267262 3403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:04.268069 3397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:04.268818 3397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:04.270059 3397 hybrid_clock.cc:648] HybridClock initialized: now 1790453764270039 us; error 39 us; skew 500 ppm
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453764, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:04.273978 3397 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:16:04.275198 3397 webserver.cc:460] Webserver started at http://127.31.119.193:40281/ using document root <none> and password file <none>
I20260926 20:16:04.275871 3397 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:04.275976 3397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:04.276253 3397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:04.278746 3397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "82cdf030bd9f49ac804c855cb28bda1d"
format_stamp: "Formatted at 2026-09-26 20:16:04 on dist-test-slave-vnbh"
server_key: "fe36932642625a7858ce3df1fb70d68e"
server_key_iv: "5f36fdbc55601888589b750cc04243b2"
server_key_version: "encryptionkey@0"
I20260926 20:16:04.279379 3397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "82cdf030bd9f49ac804c855cb28bda1d"
format_stamp: "Formatted at 2026-09-26 20:16:04 on dist-test-slave-vnbh"
server_key: "fe36932642625a7858ce3df1fb70d68e"
server_key_iv: "5f36fdbc55601888589b750cc04243b2"
server_key_version: "encryptionkey@0"
I20260926 20:16:04.283565 3397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:16:04.286448 3412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:04.287891 3397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:04.288049 3397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "82cdf030bd9f49ac804c855cb28bda1d"
format_stamp: "Formatted at 2026-09-26 20:16:04 on dist-test-slave-vnbh"
server_key: "fe36932642625a7858ce3df1fb70d68e"
server_key_iv: "5f36fdbc55601888589b750cc04243b2"
server_key_version: "encryptionkey@0"
I20260926 20:16:04.288188 3397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:04.314685 3397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:04.318282 3397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:04.318502 3397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:04.319147 3397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:04.320044 3397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:04.320115 3397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:04.320199 3397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:04.320241 3397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:04.329880 3397 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:37883
I20260926 20:16:04.329947 3526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:37883 every 8 connection(s)
I20260926 20:16:04.331296 3397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:04.334441 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 3397
I20260926 20:16:04.334592 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:16:04.334980 32223 external_mini_cluster.cc:1428] Setting key d41cb90c6848705272e417dbd15afca4
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453764, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
I20260926 20:16:04.349362 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:04.335838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:36435 (local address 127.31.119.254:40849)
0926 20:16:04.336605 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:04.336610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:04.336627 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:16:04.337384 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:04.337390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:04.337456 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:04.337558 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:04.339318 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.339925 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.341473 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.341888 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.341983 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:04.345266 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:04.345269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:04.345271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:04.345337 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:04.347008 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.347584 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.347588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.347590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.347736 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.348133 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.348136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.348137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.348332 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:04.348448 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:04.349019 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 20:16:04.349162 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674}
I20260926 20:16:04.350418 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:04.334413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40849 (local address 127.31.119.193:36435)
0926 20:16:04.335174 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:04.335262 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:04.337076 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:04.337806 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:04.337820 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:04.338447 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:04.339138 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:04.339154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.340079 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.340083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:04.341242 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:04.341260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.342235 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.342239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:04.342410 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:04.343182 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:04.343201 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:04.344996 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:04.347166 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:04.347172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:04.347189 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:04.347475 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:04.347823 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:04.347826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:04.347828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:04.348042 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:04.348445 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:04.348452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:04.348765 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 20:16:04.348969 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 20:16:04.349364 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":189,"threads_started":1}
I20260926 20:16:04.351814 3527 heartbeater.cc:347] Connected to a master server at 127.31.119.254:40849
I20260926 20:16:04.352102 3527 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:04.352646 3527 heartbeater.cc:511] Master 127.31.119.254:40849 requested a full tablet report, sending...
I20260926 20:16:04.354293 3335 ts_manager.cc:194] Registered new tserver with Master: 82cdf030bd9f49ac804c855cb28bda1d (127.31.119.193:37883)
I20260926 20:16:04.355445 3335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:36435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:16:04.365931 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40849
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:04.486653 3532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:04.486944 3532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:04.486989 3532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:04.491271 3532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:04.491338 3532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:04.491444 3532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:04.497200 3532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:04.498405 3532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:04.499524 3532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:04.509764 3538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:04.509790 3537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:04.509799 3541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:04.510119 3532 server_base.cc:1034] running on GCE node
I20260926 20:16:04.510553 3532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:04.511260 3532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:04.512437 3532 hybrid_clock.cc:648] HybridClock initialized: now 1790453764512365 us; error 82 us; skew 500 ppm
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453764, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:04.515434 3532 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:16:04.516363 3532 webserver.cc:460] Webserver started at http://127.31.119.194:43891/ using document root <none> and password file <none>
I20260926 20:16:04.516960 3532 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:04.517052 3532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:04.517343 3532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:04.519482 3532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "eef9ef81a65b442e9f312a70547d8662"
format_stamp: "Formatted at 2026-09-26 20:16:04 on dist-test-slave-vnbh"
server_key: "4dfba26d8e513e7a08b0a63405b3c170"
server_key_iv: "19098c506a9607cb540a8db0ecda857b"
server_key_version: "encryptionkey@0"
I20260926 20:16:04.520092 3532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "eef9ef81a65b442e9f312a70547d8662"
format_stamp: "Formatted at 2026-09-26 20:16:04 on dist-test-slave-vnbh"
server_key: "4dfba26d8e513e7a08b0a63405b3c170"
server_key_iv: "19098c506a9607cb540a8db0ecda857b"
server_key_version: "encryptionkey@0"
I20260926 20:16:04.524088 3532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:16:04.526841 3548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:04.528335 3532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:16:04.528568 3532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "eef9ef81a65b442e9f312a70547d8662"
format_stamp: "Formatted at 2026-09-26 20:16:04 on dist-test-slave-vnbh"
server_key: "4dfba26d8e513e7a08b0a63405b3c170"
server_key_iv: "19098c506a9607cb540a8db0ecda857b"
server_key_version: "encryptionkey@0"
I20260926 20:16:04.528712 3532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:04.563211 3532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:04.568207 3532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:04.568431 3532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:04.569108 3532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:04.570089 3532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:04.570140 3532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:04.570186 3532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:04.570242 3532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:04.580590 3662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:39719 every 8 connection(s)
I20260926 20:16:04.580629 3532 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:39719
I20260926 20:16:04.581997 3532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:04.583735 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 3532
I20260926 20:16:04.583875 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:04.584162 32223 external_mini_cluster.cc:1428] Setting key 67d18847a47b1450229a8c1e2f99eb5a
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453764, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
I20260926 20:16:04.599901 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:04.584616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:55325 (local address 127.31.119.254:40849)
0926 20:16:04.584818 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:04.584824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:04.585812 (+ 988us) server_negotiation.cc:407] Connection header received
0926 20:16:04.588042 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:04.588047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:04.588114 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:04.588228 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:04.589890 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.590507 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.591926 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.592240 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.592375 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:04.595241 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:04.595244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:04.595246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:04.595273 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:04.597061 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.597883 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.597887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.597889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.598000 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.598462 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.598468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.598473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.598846 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:04.598963 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:04.599586 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 20:16:04.599763 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260926 20:16:04.601183 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:04.584897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40849 (local address 127.31.119.194:55325)
0926 20:16:04.585625 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:04.585677 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:04.587691 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:04.588409 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:04.588417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:04.588813 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:04.589699 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:04.589723 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.590743 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.590747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:04.591781 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:04.591792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.592427 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.592431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:04.592715 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:04.593505 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:04.593524 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:04.595017 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:04.597209 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:04.597217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:04.597251 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:04.597720 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:04.598125 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:04.598131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:04.598136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:04.598333 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:04.598971 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:04.598977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:04.599284 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 20:16:04.599457 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 20:16:04.599920 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":184,"threads_started":1}
I20260926 20:16:04.602695 3663 heartbeater.cc:347] Connected to a master server at 127.31.119.254:40849
I20260926 20:16:04.603024 3663 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:04.603731 3663 heartbeater.cc:511] Master 127.31.119.254:40849 requested a full tablet report, sending...
I20260926 20:16:04.605170 3335 ts_manager.cc:194] Registered new tserver with Master: eef9ef81a65b442e9f312a70547d8662 (127.31.119.194:39719)
I20260926 20:16:04.606109 3335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:55325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:16:04.617596 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40849
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:04.734503 3668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:04.734987 3668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:04.735059 3668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:04.740186 3668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:04.740266 3668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:04.740330 3668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:04.746096 3668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:04.747184 3668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:04.748188 3668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:04.759122 3676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:04.759002 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:04.759002 3673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:04.759368 3668 server_base.cc:1034] running on GCE node
I20260926 20:16:04.759876 3668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:04.760645 3668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:04.761852 3668 hybrid_clock.cc:648] HybridClock initialized: now 1790453764761833 us; error 33 us; skew 500 ppm
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453764, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:04.765177 3668 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:16:04.766235 3668 webserver.cc:460] Webserver started at http://127.31.119.195:45569/ using document root <none> and password file <none>
I20260926 20:16:04.766842 3668 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:04.766940 3668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:04.767194 3668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:04.769373 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "02328e9fd3174cf892f8a8bb338a8f25"
format_stamp: "Formatted at 2026-09-26 20:16:04 on dist-test-slave-vnbh"
server_key: "457ff59050b379e00c336715965c230b"
server_key_iv: "249d0452618014f235f6cbc6f7425a4a"
server_key_version: "encryptionkey@0"
I20260926 20:16:04.769987 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "02328e9fd3174cf892f8a8bb338a8f25"
format_stamp: "Formatted at 2026-09-26 20:16:04 on dist-test-slave-vnbh"
server_key: "457ff59050b379e00c336715965c230b"
server_key_iv: "249d0452618014f235f6cbc6f7425a4a"
server_key_version: "encryptionkey@0"
I20260926 20:16:04.774219 3668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:16:04.777100 3683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:04.778508 3668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:16:04.778646 3668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "02328e9fd3174cf892f8a8bb338a8f25"
format_stamp: "Formatted at 2026-09-26 20:16:04 on dist-test-slave-vnbh"
server_key: "457ff59050b379e00c336715965c230b"
server_key_iv: "249d0452618014f235f6cbc6f7425a4a"
server_key_version: "encryptionkey@0"
I20260926 20:16:04.778769 3668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:04.792414 3668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:04.795918 3668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:04.796149 3668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:04.796766 3668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:04.797709 3668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:04.797755 3668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:04.797820 3668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:04.797855 3668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:04.806335 3668 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:38369
I20260926 20:16:04.806401 3797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:38369 every 8 connection(s)
I20260926 20:16:04.807291 3668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:04.814970 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 3668
I20260926 20:16:04.815132 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:04.815462 32223 external_mini_cluster.cc:1428] Setting key 6f55dfba7a9953ca26194d3fbc760921
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453764, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:04.822041 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:04.809201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:34449 (local address 127.31.119.254:40849)
0926 20:16:04.809377 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:04.809381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:04.810248 (+ 867us) server_negotiation.cc:407] Connection header received
0926 20:16:04.811680 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:04.811684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:04.811745 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:04.811817 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:04.813053 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.813696 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.814975 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.815300 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.815397 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:04.818442 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:04.818445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:04.818447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:04.818472 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:04.819850 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.820409 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.820418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.820419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.820496 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.820853 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.820857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.820859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.821046 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:04.821153 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:04.821748 (+ 595us) server_negotiation.cc:299] Negotiation successful
0926 20:16:04.821877 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:16:04.823102 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:04.809632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40849 (local address 127.31.119.195:34449)
0926 20:16:04.810087 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:04.810132 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:04.811449 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:04.812013 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:04.812020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:04.812394 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:04.812898 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:04.812912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.813823 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.813827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:04.814810 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:04.814824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.815467 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.815474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:04.815745 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:04.816407 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:04.816432 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:04.818265 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:04.819992 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:04.819998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:04.820016 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:04.820297 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:04.820600 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:04.820603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:04.820605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:04.820755 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:04.821148 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:04.821154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:04.821474 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 20:16:04.821694 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 20:16:04.822052 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":123,"threads_started":1}
I20260926 20:16:04.824321 3798 heartbeater.cc:347] Connected to a master server at 127.31.119.254:40849
I20260926 20:16:04.824609 3798 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:04.825147 3798 heartbeater.cc:511] Master 127.31.119.254:40849 requested a full tablet report, sending...
I20260926 20:16:04.826479 3335 ts_manager.cc:194] Registered new tserver with Master: 02328e9fd3174cf892f8a8bb338a8f25 (127.31.119.195:38369)
I20260926 20:16:04.827091 3335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:34449
I20260926 20:16:04.829936 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:16:04.858124 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:04.847796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.31.119.193:37883)
0926 20:16:04.848133 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:04.848148 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:04.848205 (+ 57us) server_negotiation.cc:407] Connection header received
0926 20:16:04.848260 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:04.848265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:04.848490 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:04.848618 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:04.849377 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.850195 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.851065 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.851478 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.851564 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:04.854274 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:04.854284 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:04.854299 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:04.854326 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:04.855832 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.856305 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.856312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.856313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.856404 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.856768 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.856771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.856772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.857071 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:04.857255 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:04.857740 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 20:16:04.857904 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":113,"threads_started":1}
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:04.870162 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:04.861092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.31.119.254:40849)
0926 20:16:04.861247 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:04.861250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:04.861265 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:04.861446 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:04.861449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:04.861493 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:04.861564 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:04.862281 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.863025 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.863864 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:04.864096 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:04.864184 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:04.866752 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:04.866755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:04.866757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:04.866784 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:04.868158 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.868569 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.868572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.868573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.868659 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:04.869053 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:04.869059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:04.869061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:04.869409 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:04.869535 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:04.869790 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 20:16:04.869971 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 20:16:04.870632 3335 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48132
I20260926 20:16:04.871364 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 3397
I20260926 20:16:04.876977 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 3532
I20260926 20:16:04.882762 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 3668
I20260926 20:16:04.888288 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 3303
2026-09-26T20:16:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2265 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3810](info): set up 2 sockets
Sep 26 20:16:04 dist-test-slave-vnbh krb5kdc[3810](info): commencing operation
krb5kdc: starting...
W20260926 20:16:05.946911 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:06 dist-test-slave-vnbh krb5kdc[3810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:16:06.082276 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:34989
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:06.199533 3823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:06.199878 3823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:06.199931 3823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:06.204219 3823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:06.204303 3823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:06.204324 3823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:06.204341 3823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:06.204357 3823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:06.210318 3823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:34989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:34989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:06.211530 3823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:06.212579 3823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:06.222528 3823 server_base.cc:1034] running on GCE node
W20260926 20:16:06.222492 3828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:06.222483 3831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:06.222476 3829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:06.223541 3823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:06.224821 3823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:06.226032 3823 hybrid_clock.cc:648] HybridClock initialized: now 1790453766225977 us; error 72 us; skew 500 ppm
Sep 26 20:16:06 dist-test-slave-vnbh krb5kdc[3810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:06.236521 3823 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:16:06.237576 3823 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:06.237682 3823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:06.237936 3823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:06.240267 3823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "e3e539c46a3d409fa2a7378ee281c3c7"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "00cbcb35a77736a71e147a4c66abb106"
server_key_iv: "3423bb69b3856f1d11d5fab426ff3ac9"
server_key_version: "encryptionkey@0"
I20260926 20:16:06.240877 3823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "e3e539c46a3d409fa2a7378ee281c3c7"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "00cbcb35a77736a71e147a4c66abb106"
server_key_iv: "3423bb69b3856f1d11d5fab426ff3ac9"
server_key_version: "encryptionkey@0"
I20260926 20:16:06.245152 3823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:16:06.247934 3837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:06.249658 3823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:06.249821 3823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "e3e539c46a3d409fa2a7378ee281c3c7"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "00cbcb35a77736a71e147a4c66abb106"
server_key_iv: "3423bb69b3856f1d11d5fab426ff3ac9"
server_key_version: "encryptionkey@0"
I20260926 20:16:06.249957 3823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:06.270205 3823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:06.274253 3823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:06.274504 3823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:06.282536 3823 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:34989
I20260926 20:16:06.282819 3889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:34989 every 8 connection(s)
I20260926 20:16:06.283313 3823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:06.286414 3890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:06.289808 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 3823
I20260926 20:16:06.290007 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:06.290307 32223 external_mini_cluster.cc:1428] Setting key 2ae1e11f8d5d1c8d343e50664c819b2c
I20260926 20:16:06.292842 3890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7: Bootstrap starting.
I20260926 20:16:06.295212 3890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:06.295993 3890 log.cc:824] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:06.298153 3890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7: No bootstrap required, opened a new log
Sep 26 20:16:06 dist-test-slave-vnbh krb5kdc[3810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:06.303318 3890 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e539c46a3d409fa2a7378ee281c3c7" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34989 } }
I20260926 20:16:06.303666 3890 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:06.303772 3890 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e539c46a3d409fa2a7378ee281c3c7, State: Initialized, Role: FOLLOWER
I20260926 20:16:06.304315 3890 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e539c46a3d409fa2a7378ee281c3c7" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34989 } }
I20260926 20:16:06.304486 3890 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:06.304556 3890 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:06.304665 3890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:06.305783 3890 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e539c46a3d409fa2a7378ee281c3c7" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34989 } }
I20260926 20:16:06.306185 3890 leader_election.cc:302] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3e539c46a3d409fa2a7378ee281c3c7; no voters:
I20260926 20:16:06.306533 3890 leader_election.cc:288] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:06.306676 3895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:06.306902 3895 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [term 1 LEADER]: Becoming Leader. State: Replica: e3e539c46a3d409fa2a7378ee281c3c7, State: Running, Role: LEADER
I20260926 20:16:06.307246 3895 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e539c46a3d409fa2a7378ee281c3c7" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34989 } }
I20260926 20:16:06.307534 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:06.291900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.31.119.254:34989)
0926 20:16:06.292399 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:06.292418 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:06.292468 (+ 50us) server_negotiation.cc:407] Connection header received
0926 20:16:06.293269 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:06.293316 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:06.293742 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:06.294216 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:06.295115 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:06.296379 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:06.297628 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:06.298218 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:06.298417 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:06.301272 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:06.301280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:06.301335 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:06.301379 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:06.303493 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:06.304163 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:06.304169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:06.304174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:06.304304 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:06.304744 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:06.304748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:06.304750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:06.305259 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:06.305579 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:06.305859 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 20:16:06.306242 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":156,"threads_started":1}
I20260926 20:16:06.307960 3890 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:06.308636 3895 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3e539c46a3d409fa2a7378ee281c3c7. Latest consensus state: current_term: 1 leader_uuid: "e3e539c46a3d409fa2a7378ee281c3c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e539c46a3d409fa2a7378ee281c3c7" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34989 } } }
I20260926 20:16:06.308755 3895 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:06.309060 3896 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3e539c46a3d409fa2a7378ee281c3c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e539c46a3d409fa2a7378ee281c3c7" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34989 } } }
I20260926 20:16:06.309173 3896 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:06.310007 3901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:06.313266 3901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:06.319192 3901 catalog_manager.cc:1347] Generated new cluster ID: ba713a92bc144e2cad88f4a52506ed3d
I20260926 20:16:06.319276 3901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:06.327762 3901 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:06.328769 3901 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:06.333830 3901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7: Generated new TSK 0
I20260926 20:16:06.334733 3901 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:16:06.369010 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34989
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:06.488585 3916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:06.488899 3916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:06.488948 3916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:06.493342 3916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:06.493427 3916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:06.493536 3916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:06.499892 3916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:06.501173 3916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:06.502372 3916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:06.513183 3921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:06.513183 3922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:06.513183 3924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:06.513581 3916 server_base.cc:1034] running on GCE node
I20260926 20:16:06.514175 3916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:06.514946 3916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:06.516151 3916 hybrid_clock.cc:648] HybridClock initialized: now 1790453766516111 us; error 55 us; skew 500 ppm
Sep 26 20:16:06 dist-test-slave-vnbh krb5kdc[3810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:06.519214 3916 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:16:06.520099 3916 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:06.520202 3916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:06.520452 3916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:06.522730 3916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "8c1e992c498b411cb30a539d6423f0c3"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "66d68a9e9b47b078a4a3d37dee13706f"
server_key_iv: "a998cb4e5676ab0d51df92b7d8dac35b"
server_key_version: "encryptionkey@0"
I20260926 20:16:06.523352 3916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "8c1e992c498b411cb30a539d6423f0c3"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "66d68a9e9b47b078a4a3d37dee13706f"
server_key_iv: "a998cb4e5676ab0d51df92b7d8dac35b"
server_key_version: "encryptionkey@0"
I20260926 20:16:06.527652 3916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:16:06.530362 3930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:06.531890 3916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:06.532073 3916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "8c1e992c498b411cb30a539d6423f0c3"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "66d68a9e9b47b078a4a3d37dee13706f"
server_key_iv: "a998cb4e5676ab0d51df92b7d8dac35b"
server_key_version: "encryptionkey@0"
I20260926 20:16:06.532219 3916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:06.558130 3916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:06.562386 3916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:06.562620 3916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:06.563472 3916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:06.564740 3916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:06.564829 3916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:06.564935 3916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:06.564971 3916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:06.575891 3916 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:44585
I20260926 20:16:06.575908 4044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:44585 every 8 connection(s)
I20260926 20:16:06.576604 3916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:06.586537 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 3916
I20260926 20:16:06.586658 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
Sep 26 20:16:06 dist-test-slave-vnbh krb5kdc[3810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:06.587023 32223 external_mini_cluster.cc:1428] Setting key 4cfca0b4b16d9a528e89f957c4395a45
I20260926 20:16:06.592080 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:06.578665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:40129 (local address 127.31.119.254:34989)
0926 20:16:06.578827 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:06.578833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:06.579790 (+ 957us) server_negotiation.cc:407] Connection header received
0926 20:16:06.581239 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:06.581242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:06.581328 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:06.581441 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:06.582701 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:06.583239 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:06.584427 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:06.584674 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:06.584796 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:06.587694 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:06.587698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:06.587700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:06.587731 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:06.589722 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:06.590369 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:06.590373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:06.590377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:06.590462 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:06.590936 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:06.590939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:06.590940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:06.591137 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:06.591249 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:06.591776 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 20:16:06.591910 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:16:06.592998 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:06.579033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:34989 (local address 127.31.119.193:40129)
0926 20:16:06.579589 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:06.579637 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:06.580994 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:06.581632 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:06.581640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:06.582021 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:06.582534 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:06.582548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:06.583369 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:06.583373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:06.584294 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:06.584303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:06.584841 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:06.584845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:06.585022 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:06.585723 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:06.585748 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:06.587478 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:06.589870 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:06.589876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:06.589891 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:06.590204 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:06.590639 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:06.590644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:06.590646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:06.590810 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:06.591247 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:06.591255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:06.591536 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 20:16:06.591736 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 20:16:06.592114 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":129,"threads_started":1}
I20260926 20:16:06.594461 4045 heartbeater.cc:347] Connected to a master server at 127.31.119.254:34989
I20260926 20:16:06.594743 4045 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:06.595228 4045 heartbeater.cc:511] Master 127.31.119.254:34989 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
I20260926 20:16:06.597090 3854 ts_manager.cc:194] Registered new tserver with Master: 8c1e992c498b411cb30a539d6423f0c3 (127.31.119.193:44585)
I20260926 20:16:06.598490 3854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:40129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:16:06.618721 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34989
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:06.734912 4050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:06.735206 4050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:06.735255 4050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:06.739665 4050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:06.739743 4050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:06.739809 4050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:06.746732 4050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:06.747999 4050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:06.749096 4050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:06.759569 4055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:06.759569 4058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:06.759644 4056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:06.760071 4050 server_base.cc:1034] running on GCE node
I20260926 20:16:06.760648 4050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:06.761487 4050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:06.762713 4050 hybrid_clock.cc:648] HybridClock initialized: now 1790453766762690 us; error 41 us; skew 500 ppm
Sep 26 20:16:06 dist-test-slave-vnbh krb5kdc[3810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:06.765903 4050 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:16:06.766711 4050 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:06.766780 4050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:06.766960 4050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:06.769125 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "aaac681d61ae4fb797e07c7049401bac"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "1703a6a9139f33dd97a03074bc71eede"
server_key_iv: "e5cf03c832d5895c0e8dfbdaf9a2729c"
server_key_version: "encryptionkey@0"
I20260926 20:16:06.769883 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "aaac681d61ae4fb797e07c7049401bac"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "1703a6a9139f33dd97a03074bc71eede"
server_key_iv: "e5cf03c832d5895c0e8dfbdaf9a2729c"
server_key_version: "encryptionkey@0"
I20260926 20:16:06.773860 4050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:16:06.776466 4064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:06.778043 4050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:06.778160 4050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "aaac681d61ae4fb797e07c7049401bac"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "1703a6a9139f33dd97a03074bc71eede"
server_key_iv: "e5cf03c832d5895c0e8dfbdaf9a2729c"
server_key_version: "encryptionkey@0"
I20260926 20:16:06.778261 4050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:06.802572 4050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:06.806344 4050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:06.806536 4050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:06.807156 4050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:06.808166 4050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:06.808220 4050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:06.808285 4050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:06.808318 4050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:06.817657 4050 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:33215
I20260926 20:16:06.817698 4178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:33215 every 8 connection(s)
I20260926 20:16:06.818320 4050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:06.826037 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 4050
I20260926 20:16:06.826161 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:06.826491 32223 external_mini_cluster.cc:1428] Setting key 3d298c8339b519f7bd8a1a5e965bc4f4
Sep 26 20:16:06 dist-test-slave-vnbh krb5kdc[3810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:06.833676 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:06.820247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:35467 (local address 127.31.119.254:34989)
0926 20:16:06.820387 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:06.820390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:06.821333 (+ 943us) server_negotiation.cc:407] Connection header received
0926 20:16:06.822701 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:06.822704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:06.822753 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:06.822830 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:06.824057 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:06.824692 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:06.825948 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:06.826297 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:06.826423 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:06.829837 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:06.829841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:06.829844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:06.829879 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:06.831407 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:06.831975 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:06.831979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:06.831981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:06.832060 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:06.832438 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:06.832441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:06.832443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:06.832642 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:06.832763 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:06.833345 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 20:16:06.833485 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:16:06.834631 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:06.820658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:34989 (local address 127.31.119.194:35467)
0926 20:16:06.821114 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:06.821158 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:06.822486 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:06.822995 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:06.823002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:06.823393 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:06.823874 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:06.823886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:06.824813 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:06.824816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:06.825827 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:06.825836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:06.826455 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:06.826457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:06.826654 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:06.827805 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:06.827822 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:06.829646 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:06.831539 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:06.831546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:06.831565 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:06.831852 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:06.832161 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:06.832165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:06.832174 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:06.832331 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:06.832760 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:06.832767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:06.833050 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 20:16:06.833256 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 20:16:06.833644 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
I20260926 20:16:06.835937 4179 heartbeater.cc:347] Connected to a master server at 127.31.119.254:34989
I20260926 20:16:06.836251 4179 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
I20260926 20:16:06.836796 4179 heartbeater.cc:511] Master 127.31.119.254:34989 requested a full tablet report, sending...
I20260926 20:16:06.838150 3854 ts_manager.cc:194] Registered new tserver with Master: aaac681d61ae4fb797e07c7049401bac (127.31.119.194:33215)
I20260926 20:16:06.839057 3854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:35467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:16:06.857555 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34989
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:06.974514 4184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:06.974799 4184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:06.974845 4184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:06.978956 4184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:06.979029 4184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:06.979094 4184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:06.984872 4184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:06.986023 4184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:06.987021 4184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:06.996858 4192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:06.996858 4190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:06.996976 4189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:06.997143 4184 server_base.cc:1034] running on GCE node
I20260926 20:16:06.997757 4184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:06.998910 4184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:07.000164 4184 hybrid_clock.cc:648] HybridClock initialized: now 1790453767000152 us; error 34 us; skew 500 ppm
Sep 26 20:16:06 dist-test-slave-vnbh krb5kdc[3810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:07.003815 4184 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:16:07.004688 4184 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:07.004760 4184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:07.004938 4184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:07.007239 4184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "72ca86be342546b396d33d21f93a81ac"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "c2be6a9710a41ebcf51022f674e79d85"
server_key_iv: "99f8659bf31d352dffc7fd17dbdefc4f"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.007877 4184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "72ca86be342546b396d33d21f93a81ac"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "c2be6a9710a41ebcf51022f674e79d85"
server_key_iv: "99f8659bf31d352dffc7fd17dbdefc4f"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.012358 4184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:16:07.015352 4198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:07.016832 4184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:07.017017 4184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "72ca86be342546b396d33d21f93a81ac"
format_stamp: "Formatted at 2026-09-26 20:16:06 on dist-test-slave-vnbh"
server_key: "c2be6a9710a41ebcf51022f674e79d85"
server_key_iv: "99f8659bf31d352dffc7fd17dbdefc4f"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.017150 4184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:07.037437 4184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:07.041172 4184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:07.041441 4184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:07.042098 4184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:07.042975 4184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:07.043025 4184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:07.043067 4184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:07.043126 4184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:07.052536 4184 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:41299
I20260926 20:16:07.052613 4312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:41299 every 8 connection(s)
I20260926 20:16:07.053261 4184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:07.054889 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 4184
I20260926 20:16:07.055070 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:07.055413 32223 external_mini_cluster.cc:1428] Setting key e89440bd3a8e3496df3a08dc5ecdb7af
Sep 26 20:16:07 dist-test-slave-vnbh krb5kdc[3810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:07.069103 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.055285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:45135 (local address 127.31.119.254:34989)
0926 20:16:07.055438 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.055441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.056684 (+ 1243us) server_negotiation.cc:407] Connection header received
0926 20:16:07.058372 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.058375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.058422 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.058493 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:07.059787 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.060356 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.061713 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.061939 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.062033 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.064930 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:07.064933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:07.064935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:07.064963 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:07.066648 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:07.067259 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:07.067263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:07.067265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:07.067358 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:07.067738 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:07.067743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:07.067745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:07.067942 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:07.068073 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.068730 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.068905 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:16:07.070240 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.055900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:34989 (local address 127.31.119.195:45135)
0926 20:16:07.056506 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:07.056571 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:07.058163 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:07.058675 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:07.058682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:07.059117 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:07.059636 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.059650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.060485 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.060488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.061568 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.061578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.062132 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.062136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.062298 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.063085 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:07.063103 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:07.064719 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:07.066808 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:07.066816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:07.066843 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:07.067139 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:07.067465 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:07.067467 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:07.067470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:07.067637 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:07.068073 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:07.068082 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:07.068437 (+ 355us) client_negotiation.cc:769] Sending connection context
0926 20:16:07.068680 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 20:16:07.069098 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":174,"threads_started":1}
I20260926 20:16:07.071444 4313 heartbeater.cc:347] Connected to a master server at 127.31.119.254:34989
I20260926 20:16:07.071719 4313 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:07.072223 4313 heartbeater.cc:511] Master 127.31.119.254:34989 requested a full tablet report, sending...
I20260926 20:16:07.073431 3854 ts_manager.cc:194] Registered new tserver with Master: 72ca86be342546b396d33d21f93a81ac (127.31.119.195:41299)
I20260926 20:16:07.074111 3854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:45135
I20260926 20:16:07.082607 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:16:07.093370 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.084749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.31.119.254:34989)
0926 20:16:07.084904 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.084909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.085098 (+ 189us) server_negotiation.cc:407] Connection header received
0926 20:16:07.085159 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.085161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.085208 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.085377 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:07.086218 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.086915 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.087831 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.088070 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.088161 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.089874 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:07.089877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:07.089879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:07.089906 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:07.091468 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:07.091926 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:07.091929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:07.091930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:07.092009 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:07.092252 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:07.092254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:07.092256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:07.092480 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:07.092587 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.093028 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.093137 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:16:07.096585 3854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:16:07.099896 3854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:16:07.116523 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.109676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.31.119.194:33215)
0926 20:16:07.111033 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.111047 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.111075 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:16:07.111155 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.111160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.111407 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.111575 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:07.111930 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.113198 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.114862 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.115770 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.115906 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.116086 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.116233 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.116342 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1232,"thread_start_us":115,"threads_started":1}
I20260926 20:16:07.116542 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.109338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:33215 (local address 127.0.0.1:50736)
0926 20:16:07.109847 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:07.109910 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:07.110055 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:07.111586 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:07.111593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:07.111617 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:07.111789 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.111797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.113411 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.113415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.114732 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.114742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.115902 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.115905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.116028 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.116076 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 20:16:07.116199 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:16:07.116343 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":118,"threads_started":1}
I20260926 20:16:07.117107 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.109747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:44585 (local address 127.0.0.1:56552)
0926 20:16:07.110464 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:07.110478 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:07.110563 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:07.111759 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:07.111762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:07.111795 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:07.111960 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.111968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.114619 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.114623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.115457 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.115467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.116700 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.116703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.116798 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.116823 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:16:07.116890 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:16:07.116976 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":29,"spinlock_wait_cycles":9472,"thread_start_us":98,"threads_started":1}
I20260926 20:16:07.117636 4329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.110900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.31.119.193:44585)
0926 20:16:07.111178 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.111185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.111209 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:16:07.111284 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.111289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.111454 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.111612 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:07.113379 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.114507 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.115601 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.116375 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.116495 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.116628 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.117358 (+ 730us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.117476 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":117,"threads_started":1}
I20260926 20:16:07.117704 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.110217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:41299 (local address 127.0.0.1:46300)
0926 20:16:07.110798 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:07.110814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:07.110916 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:07.111759 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:07.111762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:07.111789 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:07.111962 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.111967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.114713 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.114717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.115896 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.115906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.117318 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.117322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.117408 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.117429 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:16:07.117498 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:16:07.117564 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":48,"spinlock_wait_cycles":384,"thread_start_us":96,"threads_started":1}
I20260926 20:16:07.118173 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.110373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.31.119.195:41299)
0926 20:16:07.111176 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.111187 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.111205 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:16:07.111269 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.111273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.111410 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.111575 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:07.112085 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.113214 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.116035 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.116890 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.117034 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.117161 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.117913 (+ 752us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.118026 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":151,"threads_started":1}
I20260926 20:16:07.120455 4113 tablet_service.cc:1467] Processing CreateTablet for tablet 5f78b4ea4d974b74aeac8ff749d5b950 (DEFAULT_TABLE table=test-table [id=cf58597d2a054d5f9780ad2f53f99afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:07.120484 3960 tablet_service.cc:1467] Processing CreateTablet for tablet 5f78b4ea4d974b74aeac8ff749d5b950 (DEFAULT_TABLE table=test-table [id=cf58597d2a054d5f9780ad2f53f99afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:07.120661 4247 tablet_service.cc:1467] Processing CreateTablet for tablet 5f78b4ea4d974b74aeac8ff749d5b950 (DEFAULT_TABLE table=test-table [id=cf58597d2a054d5f9780ad2f53f99afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:07.121724 3960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f78b4ea4d974b74aeac8ff749d5b950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:07.121733 4247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f78b4ea4d974b74aeac8ff749d5b950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:07.122414 4113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f78b4ea4d974b74aeac8ff749d5b950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:07.128118 4330 tablet_bootstrap.cc:492] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3: Bootstrap starting.
I20260926 20:16:07.129630 4332 tablet_bootstrap.cc:492] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac: Bootstrap starting.
I20260926 20:16:07.129942 4331 tablet_bootstrap.cc:492] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: Bootstrap starting.
I20260926 20:16:07.130831 4330 tablet_bootstrap.cc:654] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:07.131721 4331 tablet_bootstrap.cc:654] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:07.131763 4332 tablet_bootstrap.cc:654] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:07.131721 4330 log.cc:824] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:07.133100 4331 log.cc:824] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:07.133098 4332 log.cc:824] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:07.134153 4330 tablet_bootstrap.cc:492] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3: No bootstrap required, opened a new log
I20260926 20:16:07.134374 4330 ts_tablet_manager.cc:1397] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:16:07.135078 4331 tablet_bootstrap.cc:492] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: No bootstrap required, opened a new log
I20260926 20:16:07.135078 4332 tablet_bootstrap.cc:492] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac: No bootstrap required, opened a new log
I20260926 20:16:07.135277 4332 ts_tablet_manager.cc:1397] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:16:07.135303 4331 ts_tablet_manager.cc:1397] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:16:07.138207 4330 raft_consensus.cc:348] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 } } peers { permanent_uuid: "72ca86be342546b396d33d21f93a81ac" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41299 } } peers { permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 } }
I20260926 20:16:07.138468 4330 raft_consensus.cc:374] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:07.138527 4330 raft_consensus.cc:729] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e992c498b411cb30a539d6423f0c3, State: Initialized, Role: FOLLOWER
I20260926 20:16:07.138988 4330 consensus_queue.cc:260] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 } } peers { permanent_uuid: "72ca86be342546b396d33d21f93a81ac" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41299 } } peers { permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 } }
I20260926 20:16:07.139195 4332 raft_consensus.cc:348] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 } } peers { permanent_uuid: "72ca86be342546b396d33d21f93a81ac" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41299 } } peers { permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 } }
I20260926 20:16:07.139201 4331 raft_consensus.cc:348] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 } } peers { permanent_uuid: "72ca86be342546b396d33d21f93a81ac" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41299 } } peers { permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 } }
I20260926 20:16:07.139457 4332 raft_consensus.cc:374] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:07.139457 4331 raft_consensus.cc:374] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:07.139513 4332 raft_consensus.cc:729] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaac681d61ae4fb797e07c7049401bac, State: Initialized, Role: FOLLOWER
I20260926 20:16:07.139513 4331 raft_consensus.cc:729] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ca86be342546b396d33d21f93a81ac, State: Initialized, Role: FOLLOWER
I20260926 20:16:07.139950 4332 consensus_queue.cc:260] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 } } peers { permanent_uuid: "72ca86be342546b396d33d21f93a81ac" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41299 } } peers { permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 } }
I20260926 20:16:07.139964 4331 consensus_queue.cc:260] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 } } peers { permanent_uuid: "72ca86be342546b396d33d21f93a81ac" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41299 } } peers { permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 } }
I20260926 20:16:07.140061 4330 ts_tablet_manager.cc:1428] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:16:07.140443 4045 heartbeater.cc:503] Master 127.31.119.254:34989 was elected leader, sending a full tablet report...
I20260926 20:16:07.140753 4313 heartbeater.cc:503] Master 127.31.119.254:34989 was elected leader, sending a full tablet report...
I20260926 20:16:07.140965 4332 ts_tablet_manager.cc:1428] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:16:07.140970 4179 heartbeater.cc:503] Master 127.31.119.254:34989 was elected leader, sending a full tablet report...
I20260926 20:16:07.142017 4331 ts_tablet_manager.cc:1428] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:16:07.185302 4337 raft_consensus.cc:482] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:16:07.185609 4337 raft_consensus.cc:504] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 } } peers { permanent_uuid: "72ca86be342546b396d33d21f93a81ac" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41299 } } peers { permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 } }
I20260926 20:16:07.186801 4337 leader_election.cc:288] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaac681d61ae4fb797e07c7049401bac (127.31.119.194:33215), 8c1e992c498b411cb30a539d6423f0c3 (127.31.119.193:44585)
I20260926 20:16:07.190857 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.187129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:33215 (local address 127.31.119.195:59821)
0926 20:16:07.187383 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:07.187399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:07.187504 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:07.187836 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:07.187840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:07.187864 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:07.188000 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.188010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.188830 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.188836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.189711 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.189722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.190411 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.190414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.190529 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.190547 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 20:16:07.190639 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:16:07.190716 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 20:16:07.190857 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.187252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:59821 (local address 127.31.119.194:33215)
0926 20:16:07.187396 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.187401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.187607 (+ 206us) server_negotiation.cc:407] Connection header received
0926 20:16:07.187667 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.187670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.187717 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.187804 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:07.188128 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.188674 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.189848 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.190278 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.190414 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.190470 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.190657 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.190732 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:16:07.191162 4329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.187440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:59103 (local address 127.31.119.193:44585)
0926 20:16:07.187565 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.187569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.187894 (+ 325us) server_negotiation.cc:407] Connection header received
0926 20:16:07.188077 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.188080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.188133 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.188236 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:07.188499 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.188992 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.190110 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.190540 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.190687 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.190737 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.190934 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.191034 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:16:07.191442 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.187129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:44585 (local address 127.31.119.195:59103)
0926 20:16:07.187813 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:07.187826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:07.187947 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:07.188255 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:07.188258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:07.188274 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:07.188387 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.188392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.189112 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.189116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.189980 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:07.189989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.190680 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.190684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:07.190779 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.190807 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:16:07.190894 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:16:07.191278 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":282,"thread_start_us":160,"threads_started":1}
I20260926 20:16:07.191673 4133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f78b4ea4d974b74aeac8ff749d5b950" candidate_uuid: "72ca86be342546b396d33d21f93a81ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaac681d61ae4fb797e07c7049401bac" is_pre_election: true
I20260926 20:16:07.192039 4133 raft_consensus.cc:2454] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72ca86be342546b396d33d21f93a81ac in term 0.
I20260926 20:16:07.192046 3999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f78b4ea4d974b74aeac8ff749d5b950" candidate_uuid: "72ca86be342546b396d33d21f93a81ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e992c498b411cb30a539d6423f0c3" is_pre_election: true
I20260926 20:16:07.192355 3999 raft_consensus.cc:2454] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72ca86be342546b396d33d21f93a81ac in term 0.
I20260926 20:16:07.192764 4202 leader_election.cc:302] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72ca86be342546b396d33d21f93a81ac, aaac681d61ae4fb797e07c7049401bac; no voters:
I20260926 20:16:07.193135 4337 raft_consensus.cc:2781] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:16:07.193225 4337 raft_consensus.cc:482] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:16:07.193269 4337 raft_consensus.cc:3037] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:07.194356 4337 raft_consensus.cc:504] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 } } peers { permanent_uuid: "72ca86be342546b396d33d21f93a81ac" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41299 } } peers { permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 } }
I20260926 20:16:07.194722 4337 leader_election.cc:288] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [CANDIDATE]: Term 1 election: Requested vote from peers aaac681d61ae4fb797e07c7049401bac (127.31.119.194:33215), 8c1e992c498b411cb30a539d6423f0c3 (127.31.119.193:44585)
I20260926 20:16:07.195076 4133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f78b4ea4d974b74aeac8ff749d5b950" candidate_uuid: "72ca86be342546b396d33d21f93a81ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaac681d61ae4fb797e07c7049401bac"
I20260926 20:16:07.195135 3999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f78b4ea4d974b74aeac8ff749d5b950" candidate_uuid: "72ca86be342546b396d33d21f93a81ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e992c498b411cb30a539d6423f0c3"
I20260926 20:16:07.195204 4133 raft_consensus.cc:3037] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:07.195299 3999 raft_consensus.cc:3037] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:07.196309 4133 raft_consensus.cc:2454] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72ca86be342546b396d33d21f93a81ac in term 1.
I20260926 20:16:07.196390 3999 raft_consensus.cc:2454] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72ca86be342546b396d33d21f93a81ac in term 1.
I20260926 20:16:07.196632 4202 leader_election.cc:302] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72ca86be342546b396d33d21f93a81ac, aaac681d61ae4fb797e07c7049401bac; no voters:
I20260926 20:16:07.196830 4337 raft_consensus.cc:2781] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:07.197062 4337 raft_consensus.cc:686] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 1 LEADER]: Becoming Leader. State: Replica: 72ca86be342546b396d33d21f93a81ac, State: Running, Role: LEADER
I20260926 20:16:07.197466 4337 consensus_queue.cc:237] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 } } peers { permanent_uuid: "72ca86be342546b396d33d21f93a81ac" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41299 } } peers { permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 } }
I20260926 20:16:07.200708 3852 catalog_manager.cc:5495] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 72ca86be342546b396d33d21f93a81ac (127.31.119.195). New cstate: current_term: 1 leader_uuid: "72ca86be342546b396d33d21f93a81ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72ca86be342546b396d33d21f93a81ac" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 } health_report { overall_health: UNKNOWN } } }
I20260926 20:16:07.217082 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.213272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.31.119.195:41299)
0926 20:16:07.213447 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.213452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.213468 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:16:07.213598 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.213601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.213660 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.213779 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:07.214058 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.214594 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.215485 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:07.215709 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:07.215818 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:07.216194 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:07.216650 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:07.216797 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.216833 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.216896 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 20:16:07.223481 3999 raft_consensus.cc:1261] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3 [term 1 FOLLOWER]: Refusing update from remote peer 72ca86be342546b396d33d21f93a81ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:07.223481 4133 raft_consensus.cc:1261] T 5f78b4ea4d974b74aeac8ff749d5b950 P aaac681d61ae4fb797e07c7049401bac [term 1 FOLLOWER]: Refusing update from remote peer 72ca86be342546b396d33d21f93a81ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:07.224248 4340 consensus_queue.cc:1035] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e992c498b411cb30a539d6423f0c3" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:07.224515 4337 consensus_queue.cc:1035] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaac681d61ae4fb797e07c7049401bac" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:07.232223 4345 mvcc.cc:205] Tried to move back new op lower bound from 7333698630539489280 to 7333698630443503616. Current Snapshot: MvccSnapshot[applied={T|T < 7333698630539489280}]
I20260926 20:16:07.234639 4344 mvcc.cc:205] Tried to move back new op lower bound from 7333698630539489280 to 7333698630443503616. Current Snapshot: MvccSnapshot[applied={T|T < 7333698630539489280}]
I20260926 20:16:07.243075 3854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:07.243285 3854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:07.246100 3854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3e539c46a3d409fa2a7378ee281c3c7: Sending DeleteTablet for 3 replicas of tablet 5f78b4ea4d974b74aeac8ff749d5b950
I20260926 20:16:07.247035 3960 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f78b4ea4d974b74aeac8ff749d5b950 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:07 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:56552
I20260926 20:16:07.247260 4113 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f78b4ea4d974b74aeac8ff749d5b950 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:07 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:50736
I20260926 20:16:07.247606 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 3916
I20260926 20:16:07.247628 4353 tablet_replica.cc:330] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3: stopping tablet replica
I20260926 20:16:07.247802 4247 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f78b4ea4d974b74aeac8ff749d5b950 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:07 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:46300
I20260926 20:16:07.247994 4353 raft_consensus.cc:2229] T 5f78b4ea4d974b74aeac8ff749d5b950 P 8c1e992c498b411cb30a539d6423f0c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:07.249369 4355 tablet_replica.cc:330] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: stopping tablet replica
I20260926 20:16:07.249691 4355 raft_consensus.cc:2229] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:16:07.250159 4355 raft_consensus.cc:2258] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:07.252303 4355 ts_tablet_manager.cc:1905] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:16:07.254561 3840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:44585 (error 108)
I20260926 20:16:07.255290 4355 ts_tablet_manager.cc:1918] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:07.255386 4355 log.cc:1196] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/5f78b4ea4d974b74aeac8ff749d5b950
I20260926 20:16:07.255642 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.255305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:44585 (local address 127.0.0.1:56554)
0926 20:16:07.255441 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:07.255519 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 20:16:07.255707 4355 ts_tablet_manager.cc:1939] T 5f78b4ea4d974b74aeac8ff749d5b950 P 72ca86be342546b396d33d21f93a81ac: Deleting consensus metadata
I20260926 20:16:07.255728 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 4050
I20260926 20:16:07.256992 3841 catalog_manager.cc:4841] TS 72ca86be342546b396d33d21f93a81ac (127.31.119.195:41299): tablet 5f78b4ea4d974b74aeac8ff749d5b950 (table test-table [id=cf58597d2a054d5f9780ad2f53f99afc]) successfully deleted
W20260926 20:16:07.257484 3840 catalog_manager.cc:4584] TS 8c1e992c498b411cb30a539d6423f0c3 (127.31.119.193:44585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f78b4ea4d974b74aeac8ff749d5b950: Network error: Client connection negotiation failed: client connection to 127.31.119.193:44585: connect: Connection refused (error 111)
I20260926 20:16:07.263634 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 4184
I20260926 20:16:07.263881 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.263565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:33215 (local address 127.0.0.1:50742)
0926 20:16:07.263727 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:07.263786 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.194:33215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:16:07.269695 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 3823
2026-09-26T20:16:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2382 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T20:16:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:07Z Disabled control of system clock
I20260926 20:16:07.301192 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:35385
--webserver_interface=127.31.119.254
--webserver_port=0
--builtin_ntp_servers=127.31.119.212:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:35385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:16:07.416920 4360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:07.417227 4360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:07.417266 4360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:07.421494 4360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:07.421573 4360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:07.421588 4360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:07.421607 4360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:07.421622 4360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:07.426690 4360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:35385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:07.427883 4360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:07.429368 4360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:07.438476 4365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:07.438526 4366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:07.438596 4368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:07.438666 4360 server_base.cc:1034] running on GCE node
I20260926 20:16:07.439474 4360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:07.440701 4360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:07.441906 4360 hybrid_clock.cc:648] HybridClock initialized: now 1790453767441874 us; error 64 us; skew 500 ppm
I20260926 20:16:07.444041 4360 webserver.cc:460] Webserver started at http://127.31.119.254:44351/ using document root <none> and password file <none>
I20260926 20:16:07.444675 4360 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:07.444774 4360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:07.445029 4360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:07.447304 4360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "701c088ed6b84a54bce34f7091c4f789"
format_stamp: "Formatted at 2026-09-26 20:16:07 on dist-test-slave-vnbh"
server_key: "3d06cfdad6b356f175c6d6c745fdde9c"
server_key_iv: "6e479632c6d58c61ee575aeb98510cac"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.447925 4360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "701c088ed6b84a54bce34f7091c4f789"
format_stamp: "Formatted at 2026-09-26 20:16:07 on dist-test-slave-vnbh"
server_key: "3d06cfdad6b356f175c6d6c745fdde9c"
server_key_iv: "6e479632c6d58c61ee575aeb98510cac"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.452054 4360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:16:07.454877 4374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:07.456195 4360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:16:07.456352 4360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "701c088ed6b84a54bce34f7091c4f789"
format_stamp: "Formatted at 2026-09-26 20:16:07 on dist-test-slave-vnbh"
server_key: "3d06cfdad6b356f175c6d6c745fdde9c"
server_key_iv: "6e479632c6d58c61ee575aeb98510cac"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.456485 4360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:07.461337 4360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:07.462004 4360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:07.462208 4360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:07.469471 4360 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:35385
I20260926 20:16:07.469498 4426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:35385 every 8 connection(s)
I20260926 20:16:07.470500 4360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:07.474020 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:07.477552 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 4360
I20260926 20:16:07.477694 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:07.477967 32223 external_mini_cluster.cc:1428] Setting key 172ce5f0fc997cdb5fecfced6fd7f4b6
I20260926 20:16:07.480880 4427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789: Bootstrap starting.
I20260926 20:16:07.483671 4427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:07.484467 4427 log.cc:824] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:07.484484 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.479583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.31.119.254:35385)
0926 20:16:07.480358 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.480374 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.480413 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:16:07.481173 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.481196 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.481789 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.482205 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:07.483095 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:07.483099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:07.483113 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:07.483125 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:07.483186 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:07.483203 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:07.483324 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.483431 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.483699 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":150,"threads_started":1}
I20260926 20:16:07.486744 4427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789: No bootstrap required, opened a new log
I20260926 20:16:07.490588 4427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c088ed6b84a54bce34f7091c4f789" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35385 } }
I20260926 20:16:07.490829 4427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:07.490866 4427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701c088ed6b84a54bce34f7091c4f789, State: Initialized, Role: FOLLOWER
I20260926 20:16:07.491308 4427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c088ed6b84a54bce34f7091c4f789" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35385 } }
I20260926 20:16:07.491436 4427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:07.491477 4427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:07.491549 4427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:07.492625 4427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c088ed6b84a54bce34f7091c4f789" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35385 } }
I20260926 20:16:07.493012 4427 leader_election.cc:302] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 701c088ed6b84a54bce34f7091c4f789; no voters:
I20260926 20:16:07.493373 4427 leader_election.cc:288] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:07.493479 4432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:07.493723 4432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [term 1 LEADER]: Becoming Leader. State: Replica: 701c088ed6b84a54bce34f7091c4f789, State: Running, Role: LEADER
I20260926 20:16:07.494043 4432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c088ed6b84a54bce34f7091c4f789" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35385 } }
I20260926 20:16:07.494663 4427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:07.495931 4434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 701c088ed6b84a54bce34f7091c4f789. Latest consensus state: current_term: 1 leader_uuid: "701c088ed6b84a54bce34f7091c4f789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c088ed6b84a54bce34f7091c4f789" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35385 } } }
I20260926 20:16:07.496066 4434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:07.496340 4433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "701c088ed6b84a54bce34f7091c4f789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c088ed6b84a54bce34f7091c4f789" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35385 } } }
I20260926 20:16:07.496428 4433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:07.496523 4441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:07.499827 4441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:07.505707 4441 catalog_manager.cc:1347] Generated new cluster ID: c7518fc1d9374e92abec891ae282fff5
I20260926 20:16:07.505793 4441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:07.517750 4441 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:07.518119 4441 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:07.530987 4441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789: Generated new TSK 0
I20260926 20:16:07.531878 4441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:16:07.540308 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35385
--builtin_ntp_servers=127.31.119.212:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:16:07.660606 4451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:07.661028 4451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:07.661078 4451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:07.665413 4451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:07.665508 4451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:07.665571 4451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:07.670708 4451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:07.671928 4451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:07.673448 4451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:07.684098 4451 server_base.cc:1034] running on GCE node
W20260926 20:16:07.684078 4457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:07.684161 4456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:07.684078 4459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:07.684907 4451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:07.685777 4451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:07.686971 4451 hybrid_clock.cc:648] HybridClock initialized: now 1790453767686943 us; error 67 us; skew 500 ppm
I20260926 20:16:07.688974 4451 webserver.cc:460] Webserver started at http://127.31.119.193:44261/ using document root <none> and password file <none>
I20260926 20:16:07.689669 4451 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:07.689769 4451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:07.690022 4451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:07.692183 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "b04ad0053f28484e8437aad6af4291ab"
format_stamp: "Formatted at 2026-09-26 20:16:07 on dist-test-slave-vnbh"
server_key: "490e83b94604ab33a9796e6db4aae048"
server_key_iv: "e1d55ec8df483c3c3220cde0200d0e70"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.692770 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "b04ad0053f28484e8437aad6af4291ab"
format_stamp: "Formatted at 2026-09-26 20:16:07 on dist-test-slave-vnbh"
server_key: "490e83b94604ab33a9796e6db4aae048"
server_key_iv: "e1d55ec8df483c3c3220cde0200d0e70"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.697093 4451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:16:07.699911 4465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:07.701480 4451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:07.701619 4451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "b04ad0053f28484e8437aad6af4291ab"
format_stamp: "Formatted at 2026-09-26 20:16:07 on dist-test-slave-vnbh"
server_key: "490e83b94604ab33a9796e6db4aae048"
server_key_iv: "e1d55ec8df483c3c3220cde0200d0e70"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.701761 4451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:07.707332 4451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:07.708055 4451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:07.708273 4451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:07.708880 4451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:07.709991 4451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:07.710069 4451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:07.710143 4451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:07.710179 4451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:07.719830 4451 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:35427
I20260926 20:16:07.719848 4579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:35427 every 8 connection(s)
I20260926 20:16:07.720855 4451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:07.727346 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 4451
I20260926 20:16:07.727538 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:16:07.727828 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.722921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:51133 (local address 127.31.119.254:35385)
0926 20:16:07.723074 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.723078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.724295 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 20:16:07.725840 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.725843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.725892 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.725999 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:07.727253 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:07.727257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:07.727260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:07.727280 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:07.727302 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:07.727311 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:07.727413 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.727607 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.727665 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:16:07.728025 32223 external_mini_cluster.cc:1428] Setting key 6324a9936c2e8119835344479e80ca62
I20260926 20:16:07.728744 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.723381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35385 (local address 127.31.119.193:51133)
0926 20:16:07.724097 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:07.724171 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:07.725586 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:07.726192 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:07.726199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:07.726299 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:07.726902 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:07.726903 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:07.726946 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:07.727024 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:07.727395 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:07.727404 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:07.727410 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:16:07.727612 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 20:16:07.727885 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":129,"threads_started":1}
I20260926 20:16:07.730125 4580 heartbeater.cc:347] Connected to a master server at 127.31.119.254:35385
I20260926 20:16:07.730293 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35385
--builtin_ntp_servers=127.31.119.212:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 20:16:07.730418 4580 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:07.730952 4580 heartbeater.cc:511] Master 127.31.119.254:35385 requested a full tablet report, sending...
I20260926 20:16:07.732825 4391 ts_manager.cc:194] Registered new tserver with Master: b04ad0053f28484e8437aad6af4291ab (127.31.119.193:35427)
W20260926 20:16:07.848804 4583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:07.849164 4583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:07.849241 4583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:07.853565 4583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:07.853672 4583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:07.853771 4583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:07.858958 4583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:07.860203 4583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:07.861738 4583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:07.871735 4589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:07.871834 4588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:07.871960 4583 server_base.cc:1034] running on GCE node
W20260926 20:16:07.871737 4591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:07.872562 4583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:07.873368 4583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:07.874608 4583 hybrid_clock.cc:648] HybridClock initialized: now 1790453767874566 us; error 55 us; skew 500 ppm
I20260926 20:16:07.876863 4583 webserver.cc:460] Webserver started at http://127.31.119.194:42239/ using document root <none> and password file <none>
I20260926 20:16:07.877663 4583 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:07.877770 4583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:07.878029 4583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:07.880375 4583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "899df0b529ec4581bc84e1c38861de37"
format_stamp: "Formatted at 2026-09-26 20:16:07 on dist-test-slave-vnbh"
server_key: "a1cf02ce87f5200a5d0f1a0b3dbcb4c1"
server_key_iv: "5d83e582f8b4836b86a9544bcf369695"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.881057 4583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "899df0b529ec4581bc84e1c38861de37"
format_stamp: "Formatted at 2026-09-26 20:16:07 on dist-test-slave-vnbh"
server_key: "a1cf02ce87f5200a5d0f1a0b3dbcb4c1"
server_key_iv: "5d83e582f8b4836b86a9544bcf369695"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.885519 4583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:16:07.888566 4598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:07.890187 4583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:07.890347 4583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "899df0b529ec4581bc84e1c38861de37"
format_stamp: "Formatted at 2026-09-26 20:16:07 on dist-test-slave-vnbh"
server_key: "a1cf02ce87f5200a5d0f1a0b3dbcb4c1"
server_key_iv: "5d83e582f8b4836b86a9544bcf369695"
server_key_version: "encryptionkey@0"
I20260926 20:16:07.890497 4583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:07.895531 4583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:07.896235 4583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:07.896442 4583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:07.897084 4583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:07.898243 4583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:07.898324 4583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:07.898392 4583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:07.898427 4583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:07.908026 4583 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:42027
I20260926 20:16:07.908070 4712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:42027 every 8 connection(s)
I20260926 20:16:07.909063 4583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:07.916637 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.911303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:51885 (local address 127.31.119.254:35385)
0926 20:16:07.911473 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:07.911478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:07.912841 (+ 1363us) server_negotiation.cc:407] Connection header received
0926 20:16:07.914445 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:07.914449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:07.914501 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:07.914586 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:07.916074 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:07.916078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:07.916080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:07.916094 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:07.916120 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:07.916128 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:07.916218 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:07.916401 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 20:16:07.916454 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:16:07.917654 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:07.911826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35385 (local address 127.31.119.194:51885)
0926 20:16:07.912638 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:07.912685 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:07.914175 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:07.915055 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:07.915063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:07.915178 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:07.915760 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:07.915761 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:07.915807 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:07.915886 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:07.916217 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:07.916223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:07.916228 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:16:07.916379 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 20:16:07.916678 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":161,"threads_started":1}
I20260926 20:16:07.918298 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 4583
I20260926 20:16:07.918417 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:07.918843 32223 external_mini_cluster.cc:1428] Setting key 8be528e4addf0a207725302117969eeb
I20260926 20:16:07.919260 4713 heartbeater.cc:347] Connected to a master server at 127.31.119.254:35385
I20260926 20:16:07.919549 4713 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:07.920014 4713 heartbeater.cc:511] Master 127.31.119.254:35385 requested a full tablet report, sending...
I20260926 20:16:07.921094 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35385
--builtin_ntp_servers=127.31.119.212:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 20:16:07.921165 4391 ts_manager.cc:194] Registered new tserver with Master: 899df0b529ec4581bc84e1c38861de37 (127.31.119.194:42027)
W20260926 20:16:08.041405 4716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:08.041724 4716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:08.041767 4716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:08.046129 4716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:08.046201 4716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:08.046293 4716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:08.051414 4716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:08.052636 4716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:08.054246 4716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:08.064750 4721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:08.064775 4722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:08.064750 4724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:08.065001 4716 server_base.cc:1034] running on GCE node
I20260926 20:16:08.065626 4716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:08.066413 4716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:08.067636 4716 hybrid_clock.cc:648] HybridClock initialized: now 1790453768067621 us; error 27 us; skew 500 ppm
I20260926 20:16:08.069908 4716 webserver.cc:460] Webserver started at http://127.31.119.195:36339/ using document root <none> and password file <none>
I20260926 20:16:08.070545 4716 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:08.070652 4716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:08.070896 4716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:08.073133 4716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "1a5568075c5c4efd8d7f6d64888d91e9"
format_stamp: "Formatted at 2026-09-26 20:16:08 on dist-test-slave-vnbh"
server_key: "46151a8aff8bb22501dc370c87735460"
server_key_iv: "889f8af5789cc8f730581442e4268bd9"
server_key_version: "encryptionkey@0"
I20260926 20:16:08.073829 4716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "1a5568075c5c4efd8d7f6d64888d91e9"
format_stamp: "Formatted at 2026-09-26 20:16:08 on dist-test-slave-vnbh"
server_key: "46151a8aff8bb22501dc370c87735460"
server_key_iv: "889f8af5789cc8f730581442e4268bd9"
server_key_version: "encryptionkey@0"
I20260926 20:16:08.078114 4716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:08.081418 4730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:08.083025 4716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:08.083236 4716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "1a5568075c5c4efd8d7f6d64888d91e9"
format_stamp: "Formatted at 2026-09-26 20:16:08 on dist-test-slave-vnbh"
server_key: "46151a8aff8bb22501dc370c87735460"
server_key_iv: "889f8af5789cc8f730581442e4268bd9"
server_key_version: "encryptionkey@0"
I20260926 20:16:08.083407 4716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:08.089150 4716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:08.089985 4716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:08.090216 4716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:08.090879 4716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:08.091976 4716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:08.092061 4716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:08.092142 4716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:08.092201 4716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:08.103150 4716 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:40379
I20260926 20:16:08.103195 4844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:40379 every 8 connection(s)
I20260926 20:16:08.104254 4716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:08.108237 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 4716
I20260926 20:16:08.108371 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:08.108661 32223 external_mini_cluster.cc:1428] Setting key 6c3f30a0d5a1980f2bf61d26ad597e4a
I20260926 20:16:08.113444 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.106568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:53893 (local address 127.31.119.254:35385)
0926 20:16:08.106733 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:08.106739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:08.108247 (+ 1508us) server_negotiation.cc:407] Connection header received
0926 20:16:08.110143 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:08.110147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:08.110197 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:08.110298 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:08.111684 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:08.111687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:08.111690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:08.111702 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:08.111722 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:08.111730 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:08.111811 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:08.113230 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0926 20:16:08.113310 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:16:08.113790 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.107064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35385 (local address 127.31.119.195:53893)
0926 20:16:08.108013 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:08.108082 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:08.109827 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:08.110639 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:08.110649 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:08.110759 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:08.111404 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:08.111406 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:08.111449 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:08.111526 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:08.111883 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:08.111891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:08.111897 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:16:08.112132 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 20:16:08.112466 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":142,"threads_started":1}
I20260926 20:16:08.115172 4845 heartbeater.cc:347] Connected to a master server at 127.31.119.254:35385
I20260926 20:16:08.115434 4845 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:08.115935 4845 heartbeater.cc:511] Master 127.31.119.254:35385 requested a full tablet report, sending...
I20260926 20:16:08.117304 4391 ts_manager.cc:194] Registered new tserver with Master: 1a5568075c5c4efd8d7f6d64888d91e9 (127.31.119.195:40379)
I20260926 20:16:08.123854 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:16:08.127872 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.126191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.31.119.254:35385)
0926 20:16:08.126342 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:08.126347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:08.126463 (+ 116us) server_negotiation.cc:407] Connection header received
0926 20:16:08.126569 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:08.126572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:08.126613 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:08.126697 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:08.127465 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:08.127468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:08.127470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:08.127482 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:08.127514 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:08.127522 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:08.127604 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:08.127717 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 20:16:08.127753 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:16:08.131163 4391 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:16:08.134325 4391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:16:08.147786 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.144138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.31.119.193:35427)
0926 20:16:08.145270 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:08.145311 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:08.145340 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:16:08.145401 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:08.145411 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:08.145668 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:08.146258 (+ 590us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:08.147207 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:08.147221 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:08.147230 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:08.147238 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:08.147269 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:08.147285 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:08.147402 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:08.147522 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:16:08.147605 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":138,"threads_started":1}
I20260926 20:16:08.147773 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.144008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35427 (local address 127.0.0.1:57082)
0926 20:16:08.144670 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:08.144726 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:08.144829 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:08.145847 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:08.145853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:08.145934 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:08.146562 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:08.146564 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:08.146583 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:08.146666 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:08.147376 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:08.147384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:08.147390 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:16:08.147501 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:16:08.147628 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":32,"thread_start_us":107,"threads_started":1}
I20260926 20:16:08.148089 4859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.144536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.31.119.194:42027)
0926 20:16:08.144846 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:08.144861 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:08.145492 (+ 631us) server_negotiation.cc:407] Connection header received
0926 20:16:08.145551 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:08.145556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:08.145684 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:08.145814 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:08.147255 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:08.147260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:08.147268 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:08.147281 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:08.147320 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:08.147353 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:08.147446 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:08.147703 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 20:16:08.147773 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
I20260926 20:16:08.148226 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.144086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:42027 (local address 127.0.0.1:43568)
0926 20:16:08.145249 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:08.145263 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:08.145376 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:08.146076 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:08.146079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:08.146092 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:08.147070 (+ 978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:08.147073 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:08.147083 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:08.147109 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:08.147483 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:08.147489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:08.147494 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:16:08.147559 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:16:08.147614 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"thread_start_us":128,"threads_started":1}
I20260926 20:16:08.148409 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.144964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:40379 (local address 127.0.0.1:35998)
0926 20:16:08.145767 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:08.145780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:08.145921 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:08.146570 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:08.146572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:08.146586 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:08.147026 (+ 440us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:08.147028 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:08.147038 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:08.147064 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:08.147443 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:08.147446 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:08.147450 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:16:08.147539 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:16:08.147588 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":88,"threads_started":1}
I20260926 20:16:08.148550 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.145157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.31.119.195:40379)
0926 20:16:08.145614 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:08.145627 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:08.145861 (+ 234us) server_negotiation.cc:407] Connection header received
0926 20:16:08.146056 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:08.146062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:08.146261 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:08.146394 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:08.147169 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:08.147178 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:08.147194 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:08.147209 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:08.147255 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:08.147291 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:08.147397 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:08.147703 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:16:08.147773 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":126,"threads_started":1}
I20260926 20:16:08.151165 4778 tablet_service.cc:1467] Processing CreateTablet for tablet c321a4ecda6a457f95925d307c5cb396 (DEFAULT_TABLE table=test-table [id=c287266f88994829b8fcf107ae4fbc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:08.151224 4513 tablet_service.cc:1467] Processing CreateTablet for tablet c321a4ecda6a457f95925d307c5cb396 (DEFAULT_TABLE table=test-table [id=c287266f88994829b8fcf107ae4fbc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:08.151364 4646 tablet_service.cc:1467] Processing CreateTablet for tablet c321a4ecda6a457f95925d307c5cb396 (DEFAULT_TABLE table=test-table [id=c287266f88994829b8fcf107ae4fbc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:08.152359 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c321a4ecda6a457f95925d307c5cb396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:08.152359 4513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c321a4ecda6a457f95925d307c5cb396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:08.152362 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c321a4ecda6a457f95925d307c5cb396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:08.159078 4862 tablet_bootstrap.cc:492] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: Bootstrap starting.
I20260926 20:16:08.160709 4864 tablet_bootstrap.cc:492] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: Bootstrap starting.
I20260926 20:16:08.160913 4863 tablet_bootstrap.cc:492] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab: Bootstrap starting.
I20260926 20:16:08.162384 4862 tablet_bootstrap.cc:654] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:08.162848 4863 tablet_bootstrap.cc:654] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:08.162947 4864 tablet_bootstrap.cc:654] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:08.163337 4862 log.cc:824] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:08.163614 4863 log.cc:824] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:08.163744 4864 log.cc:824] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:08.165400 4862 tablet_bootstrap.cc:492] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: No bootstrap required, opened a new log
I20260926 20:16:08.165477 4863 tablet_bootstrap.cc:492] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab: No bootstrap required, opened a new log
I20260926 20:16:08.165649 4862 ts_tablet_manager.cc:1397] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 20:16:08.165669 4863 ts_tablet_manager.cc:1397] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:16:08.166776 4864 tablet_bootstrap.cc:492] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: No bootstrap required, opened a new log
I20260926 20:16:08.166965 4864 ts_tablet_manager.cc:1397] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:16:08.169700 4863 raft_consensus.cc:348] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ad0053f28484e8437aad6af4291ab" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35427 } } peers { permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 } } peers { permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 } }
I20260926 20:16:08.169973 4863 raft_consensus.cc:374] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:08.170030 4863 raft_consensus.cc:729] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04ad0053f28484e8437aad6af4291ab, State: Initialized, Role: FOLLOWER
I20260926 20:16:08.170516 4863 consensus_queue.cc:260] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ad0053f28484e8437aad6af4291ab" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35427 } } peers { permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 } } peers { permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 } }
I20260926 20:16:08.170385 4862 raft_consensus.cc:348] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ad0053f28484e8437aad6af4291ab" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35427 } } peers { permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 } } peers { permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 } }
I20260926 20:16:08.170674 4862 raft_consensus.cc:374] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:08.170732 4862 raft_consensus.cc:729] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5568075c5c4efd8d7f6d64888d91e9, State: Initialized, Role: FOLLOWER
I20260926 20:16:08.171227 4580 heartbeater.cc:503] Master 127.31.119.254:35385 was elected leader, sending a full tablet report...
I20260926 20:16:08.171211 4862 consensus_queue.cc:260] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ad0053f28484e8437aad6af4291ab" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35427 } } peers { permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 } } peers { permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 } }
I20260926 20:16:08.171710 4864 raft_consensus.cc:348] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ad0053f28484e8437aad6af4291ab" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35427 } } peers { permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 } } peers { permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 } }
I20260926 20:16:08.171885 4845 heartbeater.cc:503] Master 127.31.119.254:35385 was elected leader, sending a full tablet report...
I20260926 20:16:08.171962 4864 raft_consensus.cc:374] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:08.172014 4864 raft_consensus.cc:729] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899df0b529ec4581bc84e1c38861de37, State: Initialized, Role: FOLLOWER
I20260926 20:16:08.172420 4864 consensus_queue.cc:260] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ad0053f28484e8437aad6af4291ab" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35427 } } peers { permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 } } peers { permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 } }
I20260926 20:16:08.173041 4863 ts_tablet_manager.cc:1428] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:16:08.173589 4864 ts_tablet_manager.cc:1428] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 20:16:08.173921 4862 ts_tablet_manager.cc:1428] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 20:16:08.174299 4713 heartbeater.cc:503] Master 127.31.119.254:35385 was elected leader, sending a full tablet report...
W20260926 20:16:08.222191 4581 tablet.cc:2369] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:16:08.355392 4846 tablet.cc:2369] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:16:08.410655 4714 tablet.cc:2369] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:16:08.579999 4868 raft_consensus.cc:482] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:16:08.580312 4868 raft_consensus.cc:504] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ad0053f28484e8437aad6af4291ab" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35427 } } peers { permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 } } peers { permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 } }
I20260926 20:16:08.581678 4868 leader_election.cc:288] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 899df0b529ec4581bc84e1c38861de37 (127.31.119.194:42027), 1a5568075c5c4efd8d7f6d64888d91e9 (127.31.119.195:40379)
I20260926 20:16:08.583899 4859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.582029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:59491 (local address 127.31.119.194:42027)
0926 20:16:08.582226 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:08.582231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:08.582253 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:16:08.582416 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:08.582419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:08.582478 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:08.582573 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:08.583419 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:08.583423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:08.583426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:08.583439 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:08.583460 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:08.583471 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:08.583568 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:08.583747 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:16:08.583785 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 20:16:08.583899 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.582187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:46445 (local address 127.31.119.195:40379)
0926 20:16:08.582321 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:08.582324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:08.582487 (+ 163us) server_negotiation.cc:407] Connection header received
0926 20:16:08.582665 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:08.582668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:08.582709 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:08.582771 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:08.583419 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:08.583423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:08.583426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:08.583439 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:08.583460 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:08.583471 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:08.583568 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:08.583719 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:16:08.583765 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:16:08.583926 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.581921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:42027 (local address 127.31.119.193:59491)
0926 20:16:08.582176 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:08.582193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:08.582294 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:08.582608 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:08.582612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:08.582630 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:08.583146 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:08.583148 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:08.583158 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:08.583243 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:08.583567 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:08.583590 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:08.583596 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:16:08.583690 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:16:08.583735 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260926 20:16:08.583927 4872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.582005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:40379 (local address 127.31.119.193:46445)
0926 20:16:08.582398 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:08.582412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:08.582560 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:08.582816 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:08.582822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:08.582840 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:08.583260 (+ 420us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:08.583262 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:08.583271 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:08.583296 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:08.583612 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:08.583615 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:08.583620 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:16:08.583690 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:16:08.583743 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260926 20:16:08.584823 4798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c321a4ecda6a457f95925d307c5cb396" candidate_uuid: "b04ad0053f28484e8437aad6af4291ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" is_pre_election: true
I20260926 20:16:08.584830 4666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c321a4ecda6a457f95925d307c5cb396" candidate_uuid: "b04ad0053f28484e8437aad6af4291ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899df0b529ec4581bc84e1c38861de37" is_pre_election: true
I20260926 20:16:08.585168 4798 raft_consensus.cc:2454] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b04ad0053f28484e8437aad6af4291ab in term 0.
I20260926 20:16:08.585178 4666 raft_consensus.cc:2454] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b04ad0053f28484e8437aad6af4291ab in term 0.
I20260926 20:16:08.585796 4468 leader_election.cc:302] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a5568075c5c4efd8d7f6d64888d91e9, 899df0b529ec4581bc84e1c38861de37, b04ad0053f28484e8437aad6af4291ab; no voters:
I20260926 20:16:08.586124 4868 raft_consensus.cc:2781] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:16:08.586208 4868 raft_consensus.cc:482] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:16:08.586241 4868 raft_consensus.cc:3037] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:08.587436 4868 raft_consensus.cc:504] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ad0053f28484e8437aad6af4291ab" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35427 } } peers { permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 } } peers { permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 } }
I20260926 20:16:08.587867 4868 leader_election.cc:288] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [CANDIDATE]: Term 1 election: Requested vote from peers 899df0b529ec4581bc84e1c38861de37 (127.31.119.194:42027), 1a5568075c5c4efd8d7f6d64888d91e9 (127.31.119.195:40379)
I20260926 20:16:08.588315 4666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c321a4ecda6a457f95925d307c5cb396" candidate_uuid: "b04ad0053f28484e8437aad6af4291ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "899df0b529ec4581bc84e1c38861de37"
I20260926 20:16:08.588336 4798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c321a4ecda6a457f95925d307c5cb396" candidate_uuid: "b04ad0053f28484e8437aad6af4291ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5568075c5c4efd8d7f6d64888d91e9"
I20260926 20:16:08.588472 4798 raft_consensus.cc:3037] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:08.588495 4666 raft_consensus.cc:3037] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:08.589799 4798 raft_consensus.cc:2454] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b04ad0053f28484e8437aad6af4291ab in term 1.
I20260926 20:16:08.589929 4666 raft_consensus.cc:2454] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b04ad0053f28484e8437aad6af4291ab in term 1.
I20260926 20:16:08.590142 4467 leader_election.cc:302] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a5568075c5c4efd8d7f6d64888d91e9, b04ad0053f28484e8437aad6af4291ab; no voters:
I20260926 20:16:08.590342 4868 raft_consensus.cc:2781] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:08.590579 4868 raft_consensus.cc:686] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [term 1 LEADER]: Becoming Leader. State: Replica: b04ad0053f28484e8437aad6af4291ab, State: Running, Role: LEADER
I20260926 20:16:08.590982 4868 consensus_queue.cc:237] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ad0053f28484e8437aad6af4291ab" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35427 } } peers { permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 } } peers { permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 } }
I20260926 20:16:08.595260 4389 catalog_manager.cc:5495] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab reported cstate change: term changed from 0 to 1, leader changed from <none> to b04ad0053f28484e8437aad6af4291ab (127.31.119.193). New cstate: current_term: 1 leader_uuid: "b04ad0053f28484e8437aad6af4291ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b04ad0053f28484e8437aad6af4291ab" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 } health_report { overall_health: UNKNOWN } } }
I20260926 20:16:08.714903 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.713060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.31.119.193:35427)
0926 20:16:08.713361 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:08.713365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:08.713379 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:16:08.713545 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:08.713547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:08.713615 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:08.713705 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:08.714446 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:08.714449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:08.714451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:08.714463 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:08.714482 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:08.714550 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:08.714637 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:08.714740 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 20:16:08.714780 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
I20260926 20:16:08.721472 4666 raft_consensus.cc:1261] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37 [term 1 FOLLOWER]: Refusing update from remote peer b04ad0053f28484e8437aad6af4291ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:08.721554 4798 raft_consensus.cc:1261] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9 [term 1 FOLLOWER]: Refusing update from remote peer b04ad0053f28484e8437aad6af4291ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:08.722288 4873 consensus_queue.cc:1035] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "899df0b529ec4581bc84e1c38861de37" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:08.722507 4868 consensus_queue.cc:1035] T c321a4ecda6a457f95925d307c5cb396 P b04ad0053f28484e8437aad6af4291ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a5568075c5c4efd8d7f6d64888d91e9" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:08.731714 4882 mvcc.cc:205] Tried to move back new op lower bound from 7333698636674887680 to 7333698636151513088. Current Snapshot: MvccSnapshot[applied={T|T < 7333698636674887680}]
I20260926 20:16:08.732475 4881 mvcc.cc:205] Tried to move back new op lower bound from 7333698636674887680 to 7333698636151513088. Current Snapshot: MvccSnapshot[applied={T|T < 7333698636674887680}]
I20260926 20:16:08.742730 4390 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:08.742944 4390 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:08.745628 4390 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 701c088ed6b84a54bce34f7091c4f789: Sending DeleteTablet for 3 replicas of tablet c321a4ecda6a457f95925d307c5cb396
I20260926 20:16:08.746507 4513 tablet_service.cc:1514] Processing DeleteTablet for tablet c321a4ecda6a457f95925d307c5cb396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:08 UTC) from {username='slave'} at 127.0.0.1:57082
I20260926 20:16:08.746515 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet c321a4ecda6a457f95925d307c5cb396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:08 UTC) from {username='slave'} at 127.0.0.1:43568
I20260926 20:16:08.747069 4889 tablet_replica.cc:330] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: stopping tablet replica
I20260926 20:16:08.747014 4778 tablet_service.cc:1514] Processing DeleteTablet for tablet c321a4ecda6a457f95925d307c5cb396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:08 UTC) from {username='slave'} at 127.0.0.1:35998
I20260926 20:16:08.747467 4889 raft_consensus.cc:2229] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:08.747566 4891 tablet_replica.cc:330] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: stopping tablet replica
I20260926 20:16:08.747766 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 4451
I20260926 20:16:08.747826 4889 raft_consensus.cc:2258] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:08.747877 4891 raft_consensus.cc:2229] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:08.748935 4891 raft_consensus.cc:2258] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:08.749814 4889 ts_tablet_manager.cc:1905] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:08.750505 4891 ts_tablet_manager.cc:1905] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:08.753404 4891 ts_tablet_manager.cc:1918] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:08.753429 4889 ts_tablet_manager.cc:1918] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:08.753530 4891 log.cc:1196] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/c321a4ecda6a457f95925d307c5cb396
I20260926 20:16:08.753530 4889 log.cc:1196] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/c321a4ecda6a457f95925d307c5cb396
I20260926 20:16:08.753854 4891 ts_tablet_manager.cc:1939] T c321a4ecda6a457f95925d307c5cb396 P 1a5568075c5c4efd8d7f6d64888d91e9: Deleting consensus metadata
I20260926 20:16:08.753855 4889 ts_tablet_manager.cc:1939] T c321a4ecda6a457f95925d307c5cb396 P 899df0b529ec4581bc84e1c38861de37: Deleting consensus metadata
I20260926 20:16:08.755300 4376 catalog_manager.cc:4841] TS 1a5568075c5c4efd8d7f6d64888d91e9 (127.31.119.195:40379): tablet c321a4ecda6a457f95925d307c5cb396 (table test-table [id=c287266f88994829b8fcf107ae4fbc40]) successfully deleted
I20260926 20:16:08.755324 4377 catalog_manager.cc:4841] TS 899df0b529ec4581bc84e1c38861de37 (127.31.119.194:42027): tablet c321a4ecda6a457f95925d307c5cb396 (table test-table [id=c287266f88994829b8fcf107ae4fbc40]) successfully deleted
W20260926 20:16:08.755777 4378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:35427 (error 108)
I20260926 20:16:08.755991 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 4583
I20260926 20:16:08.757925 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.757370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35427 (local address 127.0.0.1:57092)
0926 20:16:08.757745 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:08.757839 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":62,"threads_started":1}
W20260926 20:16:08.758136 4378 catalog_manager.cc:4584] TS b04ad0053f28484e8437aad6af4291ab (127.31.119.193:35427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c321a4ecda6a457f95925d307c5cb396: Network error: Client connection negotiation failed: client connection to 127.31.119.193:35427: connect: Connection refused (error 111)
I20260926 20:16:08.762141 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 4716
I20260926 20:16:08.768193 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 4360
2026-09-26T20:16:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1497 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 20:16:08.790920 32223 mini_oidc.cc:152] Starting JWKS server
I20260926 20:16:08.792559 32223 webserver.cc:460] Webserver started at https://127.0.0.1:36463/ using document root <none> and password file <none>
I20260926 20:16:08.792711 32223 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:16:08.792946 32223 webserver.cc:460] Webserver started at http://127.0.0.1:40995/ using document root <none> and password file <none>
2026-09-26T20:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:08Z Disabled control of system clock
I20260926 20:16:08.802760 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:37091
--webserver_interface=127.31.119.254
--webserver_port=0
--builtin_ntp_servers=127.31.119.212:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:37091
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36463/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/testchainca.cert with env {}
W20260926 20:16:08.918497 4899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:08.918802 4899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:08.918846 4899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:08.922973 4899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:08.923043 4899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:08.923060 4899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:08.923077 4899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:08.923092 4899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:16:08.923110 4899 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:16:08.923122 4899 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36463/jwks.json
I20260926 20:16:08.928823 4899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:37091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:37091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36463/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:08.930125 4899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:08.931154 4899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:08.940408 4899 server_base.cc:1034] running on GCE node
W20260926 20:16:08.940379 4907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:08.940376 4905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:08.940603 4904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:08.941231 4899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:08.942469 4899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:08.943780 4899 hybrid_clock.cc:648] HybridClock initialized: now 1790453768943745 us; error 55 us; skew 500 ppm
I20260926 20:16:08.946203 4899 webserver.cc:460] Webserver started at http://127.31.119.254:44409/ using document root <none> and password file <none>
I20260926 20:16:08.946877 4899 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:08.946980 4899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:08.947232 4899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:08.949643 4899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "f0a8c2871c3441f59d3aad2d771bfa31"
format_stamp: "Formatted at 2026-09-26 20:16:08 on dist-test-slave-vnbh"
server_key: "62c97c7d980cf24359254b77d08edc22"
server_key_iv: "7bcdefa9666e30352992d9877c3e3083"
server_key_version: "encryptionkey@0"
I20260926 20:16:08.950274 4899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "f0a8c2871c3441f59d3aad2d771bfa31"
format_stamp: "Formatted at 2026-09-26 20:16:08 on dist-test-slave-vnbh"
server_key: "62c97c7d980cf24359254b77d08edc22"
server_key_iv: "7bcdefa9666e30352992d9877c3e3083"
server_key_version: "encryptionkey@0"
I20260926 20:16:08.954417 4899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:16:08.957094 4913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:08.958547 4899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:16:08.958737 4899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "f0a8c2871c3441f59d3aad2d771bfa31"
format_stamp: "Formatted at 2026-09-26 20:16:08 on dist-test-slave-vnbh"
server_key: "62c97c7d980cf24359254b77d08edc22"
server_key_iv: "7bcdefa9666e30352992d9877c3e3083"
server_key_version: "encryptionkey@0"
I20260926 20:16:08.958869 4899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:08.972337 4899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:08.973084 4899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:08.973351 4899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:08.980414 4899 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:37091
I20260926 20:16:08.980420 4965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:37091 every 8 connection(s)
I20260926 20:16:08.981508 4899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:08.984807 4966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:08.989601 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 4899
I20260926 20:16:08.989745 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:08.990070 32223 external_mini_cluster.cc:1428] Setting key 48e35657b226d869730f615dfaa4f608
I20260926 20:16:08.991045 4966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31: Bootstrap starting.
I20260926 20:16:08.994113 4966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:08.994985 4966 log.cc:824] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:08.997696 4966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31: No bootstrap required, opened a new log
I20260926 20:16:09.000257 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:08.991557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.31.119.254:37091)
0926 20:16:08.992219 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:08.992234 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:08.992284 (+ 50us) server_negotiation.cc:407] Connection header received
0926 20:16:08.992945 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:08.992962 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:08.993509 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:08.993817 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:08.994353 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:08.995444 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:08.996800 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:08.997365 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:08.997491 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:08.998392 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:08.998398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:08.998412 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:08.998835 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:08.998894 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:08.998915 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:08.998993 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:08.999189 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 20:16:08.999495 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":137,"threads_started":1}
I20260926 20:16:09.002346 4966 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8c2871c3441f59d3aad2d771bfa31" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37091 } }
I20260926 20:16:09.002672 4966 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:09.002739 4966 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a8c2871c3441f59d3aad2d771bfa31, State: Initialized, Role: FOLLOWER
I20260926 20:16:09.003293 4966 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8c2871c3441f59d3aad2d771bfa31" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37091 } }
I20260926 20:16:09.003469 4966 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:09.003574 4966 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:09.003705 4966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:09.003741 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:16:09.004942 4966 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8c2871c3441f59d3aad2d771bfa31" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37091 } }
I20260926 20:16:09.005400 4966 leader_election.cc:302] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0a8c2871c3441f59d3aad2d771bfa31; no voters:
I20260926 20:16:09.005880 4966 leader_election.cc:288] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:09.006011 4971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:09.006251 4971 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [term 1 LEADER]: Becoming Leader. State: Replica: f0a8c2871c3441f59d3aad2d771bfa31, State: Running, Role: LEADER
I20260926 20:16:09.006593 4971 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8c2871c3441f59d3aad2d771bfa31" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37091 } }
I20260926 20:16:09.007282 4966 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:09.008632 4973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0a8c2871c3441f59d3aad2d771bfa31. Latest consensus state: current_term: 1 leader_uuid: "f0a8c2871c3441f59d3aad2d771bfa31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8c2871c3441f59d3aad2d771bfa31" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37091 } } }
I20260926 20:16:09.008646 4972 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0a8c2871c3441f59d3aad2d771bfa31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8c2871c3441f59d3aad2d771bfa31" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37091 } } }
I20260926 20:16:09.008819 4972 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:09.008821 4973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:09.009332 4981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:09.012965 4981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:09.019717 4981 catalog_manager.cc:1347] Generated new cluster ID: 8680c3028bba44cc976a3cc2ec8400d9
I20260926 20:16:09.019848 4981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:09.028587 4981 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:09.029968 4981 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:09.036121 4981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0a8c2871c3441f59d3aad2d771bfa31: Generated new TSK 0
I20260926 20:16:09.036917 4981 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:16:09.064548 4997 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 20:16:09.064710 4997 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:16:09.065351 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:09.047284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.31.119.254:37091)
0926 20:16:09.047427 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:09.047432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:09.047623 (+ 191us) server_negotiation.cc:407] Connection header received
0926 20:16:09.047827 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:09.047831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:09.047891 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:09.048010 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:16:09.048256 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.049107 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.050172 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.050529 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.050637 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:09.050986 (+ 349us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:16:09.064844 (+ 13858us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:16:09.065161 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38430: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 20:16:09.065461 4969 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38430: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 20:16:09.066843 4996 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:09.047151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:37091 (local address 127.0.0.1:38430)
0926 20:16:09.047540 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:09.047554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:09.047665 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:09.048032 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:09.048036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:09.048043 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:16:09.048148 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:09.048153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.049258 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.049262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:09.050048 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:09.050058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.050659 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.050662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:09.050738 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:09.050786 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:16:09.065133 (+ 14347us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:16:09.065463 (+ 330us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.119.254:37091: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.119.254:37091: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 20:16:09.067715 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 4899
2026-09-26T20:16:09Z chronyd exiting
I20260926 20:16:09.107951 32223 test_util.cc:175] -----------------------------------------------
I20260926 20:16:09.108044 32223 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453737971229-32223-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 20:16:09.109184 32223 mini_oidc.cc:152] Starting JWKS server
I20260926 20:16:09.110172 32223 webserver.cc:460] Webserver started at https://127.0.0.1:39159/ using document root <none> and password file <none>
I20260926 20:16:09.110260 32223 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:16:09.110469 32223 webserver.cc:460] Webserver started at http://127.0.0.1:44025/ using document root <none> and password file <none>
2026-09-26T20:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:09Z Disabled control of system clock
I20260926 20:16:09.120223 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:40407
--webserver_interface=127.31.119.254
--webserver_port=0
--builtin_ntp_servers=127.31.119.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:40407
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39159/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/testchainca.cert with env {}
W20260926 20:16:09.236510 5003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:09.236804 5003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:09.236845 5003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:09.240921 5003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:09.240994 5003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:09.241010 5003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:09.241029 5003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:09.241044 5003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:16:09.241061 5003 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:16:09.241074 5003 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39159/jwks.json
I20260926 20:16:09.246860 5003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:40407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:40407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39159/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:09.247992 5003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:09.248992 5003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:09.257819 5008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:09.257839 5009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:09.257892 5003 server_base.cc:1034] running on GCE node
W20260926 20:16:09.257817 5011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:09.258730 5003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:09.259964 5003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:09.261220 5003 hybrid_clock.cc:648] HybridClock initialized: now 1790453769261200 us; error 35 us; skew 500 ppm
I20260926 20:16:09.263401 5003 webserver.cc:460] Webserver started at http://127.31.119.254:43495/ using document root <none> and password file <none>
I20260926 20:16:09.264042 5003 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:09.264142 5003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:09.264362 5003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:09.266675 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "c1799e37c47d4e56b717ae1ca3238ee3"
format_stamp: "Formatted at 2026-09-26 20:16:09 on dist-test-slave-vnbh"
server_key: "50eeaeb6f29706f93a8060413b84e852"
server_key_iv: "3d85880485726f311c06c2bc0b40096c"
server_key_version: "encryptionkey@0"
I20260926 20:16:09.267280 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "c1799e37c47d4e56b717ae1ca3238ee3"
format_stamp: "Formatted at 2026-09-26 20:16:09 on dist-test-slave-vnbh"
server_key: "50eeaeb6f29706f93a8060413b84e852"
server_key_iv: "3d85880485726f311c06c2bc0b40096c"
server_key_version: "encryptionkey@0"
I20260926 20:16:09.271296 5003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:16:09.274174 5017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:09.275748 5003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:16:09.275949 5003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "c1799e37c47d4e56b717ae1ca3238ee3"
format_stamp: "Formatted at 2026-09-26 20:16:09 on dist-test-slave-vnbh"
server_key: "50eeaeb6f29706f93a8060413b84e852"
server_key_iv: "3d85880485726f311c06c2bc0b40096c"
server_key_version: "encryptionkey@0"
I20260926 20:16:09.276124 5003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:09.290504 5003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:09.291306 5003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:09.291532 5003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:09.299116 5003 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:40407
I20260926 20:16:09.299167 5069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:40407 every 8 connection(s)
I20260926 20:16:09.300172 5003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:09.303678 5070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:09.306737 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 5003
I20260926 20:16:09.306907 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:09.307210 32223 external_mini_cluster.cc:1428] Setting key 7ac4849cd8bd2cd310aa4a6b11aec278
I20260926 20:16:09.310307 5070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3: Bootstrap starting.
I20260926 20:16:09.312805 5070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:09.313681 5070 log.cc:824] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:09.315692 5070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3: No bootstrap required, opened a new log
I20260926 20:16:09.317548 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:09.308681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.31.119.254:40407)
0926 20:16:09.309462 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:09.309478 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:09.309516 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:16:09.310205 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:09.310224 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:09.310756 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:09.311134 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:09.311602 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.312632 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.313881 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.314427 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.314636 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:09.315599 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:09.315604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:09.315618 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:09.316091 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:09.316151 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:09.316173 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:09.316297 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:09.316408 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 20:16:09.316695 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":146,"threads_started":1}
I20260926 20:16:09.319890 5070 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1799e37c47d4e56b717ae1ca3238ee3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40407 } }
I20260926 20:16:09.320163 5070 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:09.320227 5070 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1799e37c47d4e56b717ae1ca3238ee3, State: Initialized, Role: FOLLOWER
I20260926 20:16:09.320736 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:16:09.320775 5070 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1799e37c47d4e56b717ae1ca3238ee3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40407 } }
I20260926 20:16:09.320950 5070 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:09.321014 5070 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:09.321113 5070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:09.322266 5070 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1799e37c47d4e56b717ae1ca3238ee3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40407 } }
I20260926 20:16:09.322623 5070 leader_election.cc:302] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1799e37c47d4e56b717ae1ca3238ee3; no voters:
I20260926 20:16:09.323019 5070 leader_election.cc:288] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:09.323175 5075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:09.323434 5075 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [term 1 LEADER]: Becoming Leader. State: Replica: c1799e37c47d4e56b717ae1ca3238ee3, State: Running, Role: LEADER
I20260926 20:16:09.323776 5075 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1799e37c47d4e56b717ae1ca3238ee3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40407 } }
I20260926 20:16:09.324357 5070 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:09.325625 5077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1799e37c47d4e56b717ae1ca3238ee3. Latest consensus state: current_term: 1 leader_uuid: "c1799e37c47d4e56b717ae1ca3238ee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1799e37c47d4e56b717ae1ca3238ee3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40407 } } }
I20260926 20:16:09.325624 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1799e37c47d4e56b717ae1ca3238ee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1799e37c47d4e56b717ae1ca3238ee3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40407 } } }
I20260926 20:16:09.325778 5077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:09.325778 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:09.326261 5084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:09.329566 5084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:09.335656 5084 catalog_manager.cc:1347] Generated new cluster ID: 354a398495fe4f17b8b90359528be574
I20260926 20:16:09.335750 5084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:09.368124 5084 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:09.369097 5084 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:09.375866 5084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1799e37c47d4e56b717ae1ca3238ee3: Generated new TSK 0
I20260926 20:16:09.376703 5084 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:16:09.401783 5100 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 20:16:09.401883 5100 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:16:09.402570 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:09.386222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.31.119.254:40407)
0926 20:16:09.386396 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:09.386402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:09.386597 (+ 195us) server_negotiation.cc:407] Connection header received
0926 20:16:09.386772 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:09.386775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:09.386829 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:09.386927 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:16:09.387186 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.387786 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.389060 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.389362 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.389451 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:09.389799 (+ 348us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:16:09.402200 (+ 12401us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:16:09.402441 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 20:16:09.402673 5073 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 20:16:09.402835 5099 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:09.386146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40407 (local address 127.0.0.1:56084)
0926 20:16:09.386487 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:09.386504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:09.386647 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:09.386950 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:09.386953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:09.386961 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:16:09.387079 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:09.387085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.387901 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.387905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:09.388904 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:09.388915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.389487 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.389491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:09.389580 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:09.389610 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:16:09.402423 (+ 12813us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:16:09.402633 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.119.254:40407: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":127,"threads_started":1}
I20260926 20:16:09.403546 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 5003
2026-09-26T20:16:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 20:16:09.443358 32223 mini_oidc.cc:152] Starting JWKS server
I20260926 20:16:09.444257 32223 webserver.cc:460] Webserver started at https://127.0.0.1:41413/ using document root <none> and password file <none>
I20260926 20:16:09.444389 32223 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:16:09.444685 32223 webserver.cc:460] Webserver started at http://127.0.0.1:46843/ using document root <none> and password file <none>
2026-09-26T20:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:09Z Disabled control of system clock
I20260926 20:16:09.453770 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:42791
--webserver_interface=127.31.119.254
--webserver_port=0
--builtin_ntp_servers=127.31.119.212:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:42791
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41413/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:16:09.567288 5106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:09.567582 5106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:09.567622 5106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:09.571789 5106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:09.571866 5106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:09.571883 5106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:09.571902 5106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:09.571916 5106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:16:09.571933 5106 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:16:09.571988 5106 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41413/jwks.json
I20260926 20:16:09.577009 5106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:42791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41413/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:09.578246 5106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:09.579708 5106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:09.589149 5111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:09.589208 5106 server_base.cc:1034] running on GCE node
W20260926 20:16:09.589164 5112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:09.589164 5114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:09.590042 5106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:09.591289 5106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:09.592482 5106 hybrid_clock.cc:648] HybridClock initialized: now 1790453769592445 us; error 56 us; skew 500 ppm
I20260926 20:16:09.594524 5106 webserver.cc:460] Webserver started at http://127.31.119.254:40143/ using document root <none> and password file <none>
I20260926 20:16:09.595091 5106 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:09.595160 5106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:09.595330 5106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:09.597528 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "6d6542543e9744d3b9bedf83a282c50a"
format_stamp: "Formatted at 2026-09-26 20:16:09 on dist-test-slave-vnbh"
server_key: "000900cdbd01fcfe3c00072d41d64baa"
server_key_iv: "949583259886e9eabd1acf28a63b7e38"
server_key_version: "encryptionkey@0"
I20260926 20:16:09.598101 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "6d6542543e9744d3b9bedf83a282c50a"
format_stamp: "Formatted at 2026-09-26 20:16:09 on dist-test-slave-vnbh"
server_key: "000900cdbd01fcfe3c00072d41d64baa"
server_key_iv: "949583259886e9eabd1acf28a63b7e38"
server_key_version: "encryptionkey@0"
I20260926 20:16:09.601938 5106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:16:09.604375 5120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:09.605865 5106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:09.605990 5106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "6d6542543e9744d3b9bedf83a282c50a"
format_stamp: "Formatted at 2026-09-26 20:16:09 on dist-test-slave-vnbh"
server_key: "000900cdbd01fcfe3c00072d41d64baa"
server_key_iv: "949583259886e9eabd1acf28a63b7e38"
server_key_version: "encryptionkey@0"
I20260926 20:16:09.606091 5106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:09.618585 5106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:09.619298 5106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:09.619463 5106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:09.626555 5106 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:42791
I20260926 20:16:09.626578 5173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:42791 every 8 connection(s)
I20260926 20:16:09.627600 5106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:09.629979 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 5106
I20260926 20:16:09.630095 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:09.630386 32223 external_mini_cluster.cc:1428] Setting key 2a232ae7972bd6d4162a2d076bfc6180
I20260926 20:16:09.631588 5174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:09.639060 5174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a: Bootstrap starting.
I20260926 20:16:09.641989 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:09.631833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.31.119.254:42791)
0926 20:16:09.632532 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:09.632553 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:09.632597 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:16:09.633514 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:09.633538 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:09.634079 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:09.634504 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:09.634998 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.636474 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.637638 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.638409 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.638590 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:09.639662 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:09.639669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:09.639687 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:09.640223 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:09.640304 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:09.640339 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:09.640452 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:09.640637 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 20:16:09.640944 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260926 20:16:09.642091 5174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:09.642858 5174 log.cc:824] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:09.645015 5174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a: No bootstrap required, opened a new log
I20260926 20:16:09.645175 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:16:09.648844 5174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6542543e9744d3b9bedf83a282c50a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42791 } }
I20260926 20:16:09.649088 5174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:09.649124 5174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6542543e9744d3b9bedf83a282c50a, State: Initialized, Role: FOLLOWER
I20260926 20:16:09.649572 5174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6542543e9744d3b9bedf83a282c50a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42791 } }
I20260926 20:16:09.649715 5174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:09.649766 5174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:09.649839 5174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:09.650813 5174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6542543e9744d3b9bedf83a282c50a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42791 } }
I20260926 20:16:09.651124 5174 leader_election.cc:302] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d6542543e9744d3b9bedf83a282c50a; no voters:
I20260926 20:16:09.651489 5174 leader_election.cc:288] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:09.651568 5179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:09.651787 5179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [term 1 LEADER]: Becoming Leader. State: Replica: 6d6542543e9744d3b9bedf83a282c50a, State: Running, Role: LEADER
I20260926 20:16:09.652093 5179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6542543e9744d3b9bedf83a282c50a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42791 } }
I20260926 20:16:09.652909 5174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:09.653357 5179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d6542543e9744d3b9bedf83a282c50a. Latest consensus state: current_term: 1 leader_uuid: "6d6542543e9744d3b9bedf83a282c50a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6542543e9744d3b9bedf83a282c50a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42791 } } }
I20260926 20:16:09.653478 5179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:09.653677 5180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d6542543e9744d3b9bedf83a282c50a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6542543e9744d3b9bedf83a282c50a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42791 } } }
I20260926 20:16:09.653838 5180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:09.653905 5183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:09.657460 5183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:09.663885 5183 catalog_manager.cc:1347] Generated new cluster ID: b6e5410dbe3c40d3a9aa19621a3edc6f
I20260926 20:16:09.663980 5183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:09.682302 5183 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:09.683650 5183 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:09.700309 5183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d6542543e9744d3b9bedf83a282c50a: Generated new TSK 0
I20260926 20:16:09.701273 5183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:16:09.716998 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:09.711564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.31.119.254:42791)
0926 20:16:09.711710 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:09.711715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:09.711938 (+ 223us) server_negotiation.cc:407] Connection header received
0926 20:16:09.712109 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:09.712111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:09.712186 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:09.712278 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:16:09.712545 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.713071 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.714369 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.714591 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.714677 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:09.715038 (+ 361us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:16:09.716619 (+ 1581us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:16:09.716852 (+ 233us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 20:16:09.717073 5177 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 20:16:09.718829 5203 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:09.711508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:42791 (local address 127.0.0.1:55248)
0926 20:16:09.711841 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:09.711858 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:09.711980 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:09.712308 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:09.712311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:09.712325 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:16:09.712438 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:09.712444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.713189 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.713193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:09.714231 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:09.714244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.714720 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.714724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:09.714817 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:09.714855 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:16:09.716799 (+ 1944us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:16:09.718641 (+ 1842us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.119.254:42791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":1616,"thread_start_us":120,"threads_started":1}
I20260926 20:16:09.719801 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 5106
E20260926 20:16:09.727882 5204 webserver.cc:533] Webserver: sslize failed:
E20260926 20:16:09.727960 5204 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T20:16:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (317 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T20:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:09Z Disabled control of system clock
I20260926 20:16:09.769593 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:33507
--webserver_interface=127.31.119.254
--webserver_port=0
--builtin_ntp_servers=127.31.119.212:43849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:33507
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/testchainca.cert with env {}
W20260926 20:16:09.884277 5208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:09.884686 5208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:09.884734 5208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:09.889031 5208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:09.889107 5208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:09.889123 5208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:09.889144 5208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:09.889160 5208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:16:09.889176 5208 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:16:09.889190 5208 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 20:16:09.894976 5208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:43849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:33507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:33507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:09.896086 5208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:09.897074 5208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:09.905874 5216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:09.905889 5214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:09.905967 5208 server_base.cc:1034] running on GCE node
W20260926 20:16:09.905874 5213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:09.906893 5208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:09.908170 5208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:09.909375 5208 hybrid_clock.cc:648] HybridClock initialized: now 1790453769909361 us; error 31 us; skew 500 ppm
I20260926 20:16:09.911737 5208 webserver.cc:460] Webserver started at http://127.31.119.254:34847/ using document root <none> and password file <none>
I20260926 20:16:09.912384 5208 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:09.912485 5208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:09.912750 5208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:09.915226 5208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "47acad8180fd4d81b741eea2ef4737b9"
format_stamp: "Formatted at 2026-09-26 20:16:09 on dist-test-slave-vnbh"
server_key: "d7e17893693d222271c95953b2e09f9d"
server_key_iv: "f5038dd9ae668b2a08cde137546fe6fc"
server_key_version: "encryptionkey@0"
I20260926 20:16:09.915877 5208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "47acad8180fd4d81b741eea2ef4737b9"
format_stamp: "Formatted at 2026-09-26 20:16:09 on dist-test-slave-vnbh"
server_key: "d7e17893693d222271c95953b2e09f9d"
server_key_iv: "f5038dd9ae668b2a08cde137546fe6fc"
server_key_version: "encryptionkey@0"
I20260926 20:16:09.920153 5208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:16:09.923147 5222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:09.924484 5208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:09.924685 5208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "47acad8180fd4d81b741eea2ef4737b9"
format_stamp: "Formatted at 2026-09-26 20:16:09 on dist-test-slave-vnbh"
server_key: "d7e17893693d222271c95953b2e09f9d"
server_key_iv: "f5038dd9ae668b2a08cde137546fe6fc"
server_key_version: "encryptionkey@0"
I20260926 20:16:09.924818 5208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:09.947129 5208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:09.947962 5208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:09.948186 5208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:09.956243 5208 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:33507
I20260926 20:16:09.956247 5274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:33507 every 8 connection(s)
I20260926 20:16:09.957403 5208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:09.960605 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:09.966085 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 5208
I20260926 20:16:09.966251 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:09.966545 32223 external_mini_cluster.cc:1428] Setting key fdcb52b9431708085be3737998cab5b7
I20260926 20:16:09.966990 5275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9: Bootstrap starting.
I20260926 20:16:09.969520 5275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:09.970361 5275 log.cc:824] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:09.972499 5275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9: No bootstrap required, opened a new log
I20260926 20:16:09.976161 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:09.968044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.31.119.254:33507)
0926 20:16:09.968779 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:09.968793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:09.968830 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:16:09.969528 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:09.969546 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:09.970007 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:09.970295 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:09.970717 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.971660 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.972853 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:09.973362 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:09.973519 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:09.974328 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:09.974333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:09.974345 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:09.974775 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:09.974828 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:09.974847 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:09.974922 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:09.975089 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 20:16:09.975376 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":150,"threads_started":1}
I20260926 20:16:09.976641 5275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47acad8180fd4d81b741eea2ef4737b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33507 } }
I20260926 20:16:09.976856 5275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:09.976914 5275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47acad8180fd4d81b741eea2ef4737b9, State: Initialized, Role: FOLLOWER
I20260926 20:16:09.977437 5275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47acad8180fd4d81b741eea2ef4737b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33507 } }
I20260926 20:16:09.977593 5275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:09.977659 5275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:09.977751 5275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:09.978892 5275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47acad8180fd4d81b741eea2ef4737b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33507 } }
I20260926 20:16:09.979332 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:16:09.979285 5275 leader_election.cc:302] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47acad8180fd4d81b741eea2ef4737b9; no voters:
I20260926 20:16:09.979617 5275 leader_election.cc:288] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:09.979753 5280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:09.980006 5280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [term 1 LEADER]: Becoming Leader. State: Replica: 47acad8180fd4d81b741eea2ef4737b9, State: Running, Role: LEADER
I20260926 20:16:09.980392 5280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47acad8180fd4d81b741eea2ef4737b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33507 } }
I20260926 20:16:09.980947 5275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:09.982277 5282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47acad8180fd4d81b741eea2ef4737b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47acad8180fd4d81b741eea2ef4737b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33507 } } }
I20260926 20:16:09.982326 5281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47acad8180fd4d81b741eea2ef4737b9. Latest consensus state: current_term: 1 leader_uuid: "47acad8180fd4d81b741eea2ef4737b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47acad8180fd4d81b741eea2ef4737b9" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33507 } } }
I20260926 20:16:09.982407 5282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:09.982420 5281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9 [sys.catalog]: This master's current role is: LEADER
W20260926 20:16:09.986071 5295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:16:09.986181 5295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:16:09.986255 5289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:09.987210 5289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:09.994580 5289 catalog_manager.cc:1347] Generated new cluster ID: 62461add8f41474d89df978215c4dab0
I20260926 20:16:09.994776 5289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:10.012176 5289 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:10.013504 5289 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:10.020584 5289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47acad8180fd4d81b741eea2ef4737b9: Generated new TSK 0
I20260926 20:16:10.021566 5289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:16:10.491427 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:10.037840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.31.119.254:33507)
0926 20:16:10.037982 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:10.037989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:10.038136 (+ 147us) server_negotiation.cc:407] Connection header received
0926 20:16:10.038389 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:10.038392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:10.038446 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:10.038572 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:16:10.038860 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:10.039434 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:10.040862 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:10.041097 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:10.041185 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:10.041758 (+ 573us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:16:10.490985 (+449227us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:16:10.491304 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52710: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 20:16:10.491540 5278 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52710: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 20:16:10.491671 5304 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:10.037656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:33507 (local address 127.0.0.1:52710)
0926 20:16:10.038029 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:10.038047 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:10.038178 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:10.038598 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:10.038604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:10.038613 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:16:10.038739 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:10.038746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:10.039568 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:10.039572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:10.040712 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:10.040724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:10.041243 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:10.041248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:10.041371 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:10.041409 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:16:10.491254 (+449845us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:16:10.491466 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.119.254:33507: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260926 20:16:10.492386 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 5208
2026-09-26T20:16:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (751 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (133 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (130 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:16:10 dist-test-slave-vnbh krb5kdc[5316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:16:10 dist-test-slave-vnbh krb5kdc[5316](info): set up 2 sockets
Sep 26 20:16:10 dist-test-slave-vnbh krb5kdc[5316](info): commencing operation
krb5kdc: starting...
W20260926 20:16:11.830777 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:11 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:16:11.961652 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:41931
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:41931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:12.077659 5329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:12.078001 5329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:12.078083 5329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:12.082182 5329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:12.082283 5329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:12.082320 5329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:12.082377 5329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:12.082415 5329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:12.087942 5329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:41931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:41931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:12.089246 5329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:12.090385 5329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:12.099609 5329 server_base.cc:1034] running on GCE node
W20260926 20:16:12.099538 5334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:12.099547 5337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:12.099642 5335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:12.100395 5329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:12.101663 5329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:12.102854 5329 hybrid_clock.cc:648] HybridClock initialized: now 1790453772102826 us; error 41 us; skew 500 ppm
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:12.105999 5329 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:16:12.106925 5329 webserver.cc:460] Webserver started at http://127.31.119.254:44903/ using document root <none> and password file <none>
I20260926 20:16:12.107481 5329 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:12.107543 5329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:12.107712 5329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:12.109977 5329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "76d7e410740a4999b13e290c52234cb3"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "4421f59e61d6cf44a30c2f6439914f56"
server_key_iv: "58ffb7f7a7cfb90d107aabdfd17972c1"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.110575 5329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "76d7e410740a4999b13e290c52234cb3"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "4421f59e61d6cf44a30c2f6439914f56"
server_key_iv: "58ffb7f7a7cfb90d107aabdfd17972c1"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.114734 5329 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:12.117626 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:12.119151 5329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:12.119330 5329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "76d7e410740a4999b13e290c52234cb3"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "4421f59e61d6cf44a30c2f6439914f56"
server_key_iv: "58ffb7f7a7cfb90d107aabdfd17972c1"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.119467 5329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:12.146298 5329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:12.150087 5329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:12.150308 5329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:12.157533 5329 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:41931
I20260926 20:16:12.157558 5396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:41931 every 8 connection(s)
I20260926 20:16:12.158578 5329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:12.158931 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 5329
I20260926 20:16:12.159085 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:12.159432 32223 external_mini_cluster.cc:1428] Setting key 6e0bdfb44bfce56e8926054e13bb657c
I20260926 20:16:12.162083 5397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:12.169088 5397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3: Bootstrap starting.
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:12.172003 5397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:12.172925 5397 log.cc:824] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:12.175390 5397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3: No bootstrap required, opened a new log
I20260926 20:16:12.175675 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:12.161113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.31.119.254:41931)
0926 20:16:12.161626 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:12.161646 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:12.161689 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:16:12.162543 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:12.162569 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:12.163008 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:12.163389 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:12.164134 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.165722 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.166628 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.167330 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.167510 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:12.170274 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:12.170283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:12.170303 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:12.170343 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:12.172509 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.172962 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.172969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.172975 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.173113 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.173446 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.173451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.173456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.173929 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:12.174115 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:12.174371 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 20:16:12.174682 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":200,"threads_started":1}
I20260926 20:16:12.179215 5397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d7e410740a4999b13e290c52234cb3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41931 } }
I20260926 20:16:12.179454 5397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:12.179489 5397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d7e410740a4999b13e290c52234cb3, State: Initialized, Role: FOLLOWER
I20260926 20:16:12.179855 5397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d7e410740a4999b13e290c52234cb3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41931 } }
I20260926 20:16:12.179965 5397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:12.180012 5397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:12.180090 5397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:12.181027 5397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d7e410740a4999b13e290c52234cb3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41931 } }
I20260926 20:16:12.181398 5397 leader_election.cc:302] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76d7e410740a4999b13e290c52234cb3; no voters:
I20260926 20:16:12.181680 5397 leader_election.cc:288] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:12.181823 5402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:12.182061 5402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 76d7e410740a4999b13e290c52234cb3, State: Running, Role: LEADER
I20260926 20:16:12.182438 5402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d7e410740a4999b13e290c52234cb3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41931 } }
I20260926 20:16:12.182943 5397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:12.184475 5404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d7e410740a4999b13e290c52234cb3. Latest consensus state: current_term: 1 leader_uuid: "76d7e410740a4999b13e290c52234cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d7e410740a4999b13e290c52234cb3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41931 } } }
I20260926 20:16:12.184625 5404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:12.185357 5414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:12.185467 5403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76d7e410740a4999b13e290c52234cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d7e410740a4999b13e290c52234cb3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41931 } } }
I20260926 20:16:12.185590 5403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:12.188180 5414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:12.194511 5414 catalog_manager.cc:1347] Generated new cluster ID: 5af3287a1880481782ef9483a843068e
I20260926 20:16:12.194604 5414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:12.210690 5414 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:12.212194 5414 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:12.219831 5414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76d7e410740a4999b13e290c52234cb3: Generated new TSK 0
I20260926 20:16:12.220613 5414 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:16:12.253080 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41931
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:12.371467 5423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:12.371765 5423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:12.371812 5423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:12.376075 5423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:12.376154 5423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:12.376245 5423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:12.381996 5423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:12.383257 5423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:12.384575 5423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:12.396209 5431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:12.396209 5429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:12.396219 5428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:12.396687 5423 server_base.cc:1034] running on GCE node
I20260926 20:16:12.397231 5423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:12.398031 5423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:12.399217 5423 hybrid_clock.cc:648] HybridClock initialized: now 1790453772399197 us; error 35 us; skew 500 ppm
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:12.402788 5423 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:16:12.403838 5423 webserver.cc:460] Webserver started at http://127.31.119.193:41083/ using document root <none> and password file <none>
I20260926 20:16:12.404492 5423 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:12.404595 5423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:12.404842 5423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:12.407208 5423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "e5ec597b686546f69d3e78c200ca932c"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "d0e6e3d4284d9000744290b7dd0964d2"
server_key_iv: "75a263ae24dc417ad1ac350d4815dbe8"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.407823 5423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "e5ec597b686546f69d3e78c200ca932c"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "d0e6e3d4284d9000744290b7dd0964d2"
server_key_iv: "75a263ae24dc417ad1ac350d4815dbe8"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.411883 5423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 20:16:12.414727 5438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:12.416090 5423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:16:12.416245 5423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "e5ec597b686546f69d3e78c200ca932c"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "d0e6e3d4284d9000744290b7dd0964d2"
server_key_iv: "75a263ae24dc417ad1ac350d4815dbe8"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.416373 5423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:12.433499 5423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:12.436857 5423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:12.437070 5423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:12.437748 5423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:12.438591 5423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:12.438661 5423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:12.438735 5423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:12.438786 5423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:12.448590 5423 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:36515
I20260926 20:16:12.448616 5551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:36515 every 8 connection(s)
I20260926 20:16:12.449607 5423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:12.450295 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 5423
I20260926 20:16:12.450426 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:16:12.450809 32223 external_mini_cluster.cc:1428] Setting key faccc9fe0267ba2a5e68ba9df7234ef8
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:12.469904 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:12.451985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:58895 (local address 127.31.119.254:41931)
0926 20:16:12.452144 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:12.452149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:12.453203 (+ 1054us) server_negotiation.cc:407] Connection header received
0926 20:16:12.454959 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:12.454963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:12.455031 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:12.455130 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:12.456768 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.457380 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.459331 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.459571 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.459666 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:12.465838 (+ 6172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:12.465842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:12.465844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:12.465876 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:12.467459 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.468146 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.468157 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.468159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.468237 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.468624 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.468629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.468631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.468832 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:12.468944 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:12.469588 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 20:16:12.469733 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:16:12.470834 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:12.452497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41931 (local address 127.31.119.193:58895)
0926 20:16:12.453025 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:12.453087 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:12.454744 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:12.455294 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:12.455301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:12.455833 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:12.456596 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:12.456620 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.457511 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.457515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:12.459180 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:12.459193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.459750 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.459755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:12.460000 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:12.460715 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:12.460740 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:12.465590 (+ 4850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:12.467661 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:12.467669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:12.467692 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:12.468005 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:12.468340 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:12.468343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:12.468345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:12.468522 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:12.468944 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:12.468954 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:12.469260 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 20:16:12.469544 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 20:16:12.469926 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260926 20:16:12.472368 5552 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41931
I20260926 20:16:12.472707 5552 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:12.473263 5552 heartbeater.cc:511] Master 127.31.119.254:41931 requested a full tablet report, sending...
I20260926 20:16:12.475101 5361 ts_manager.cc:194] Registered new tserver with Master: e5ec597b686546f69d3e78c200ca932c (127.31.119.193:36515)
I20260926 20:16:12.476532 5361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:58895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:16:12.481822 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41931
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:12.602805 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:12.603112 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:12.603159 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:12.607445 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:12.607538 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:12.607605 5557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:12.613382 5557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:12.614593 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:12.615692 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:12.626281 5563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:12.626304 5562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:12.626281 5565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:12.626626 5557 server_base.cc:1034] running on GCE node
I20260926 20:16:12.627106 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:12.627804 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:12.629045 5557 hybrid_clock.cc:648] HybridClock initialized: now 1790453772628998 us; error 60 us; skew 500 ppm
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:12.632340 5557 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:16:12.633564 5557 webserver.cc:460] Webserver started at http://127.31.119.194:40433/ using document root <none> and password file <none>
I20260926 20:16:12.634202 5557 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:12.634301 5557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:12.634552 5557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:12.636831 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "78c43f3e18dc4201b865efb76acf4b7b"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "219d4330b2cbf454b5e2f9c96d1a0ab8"
server_key_iv: "15f076a03176a1227cdeb39a11946475"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.637547 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "78c43f3e18dc4201b865efb76acf4b7b"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "219d4330b2cbf454b5e2f9c96d1a0ab8"
server_key_iv: "15f076a03176a1227cdeb39a11946475"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.641875 5557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:16:12.644796 5573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:12.646312 5557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:12.646485 5557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "78c43f3e18dc4201b865efb76acf4b7b"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "219d4330b2cbf454b5e2f9c96d1a0ab8"
server_key_iv: "15f076a03176a1227cdeb39a11946475"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.646642 5557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:12.674117 5557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:12.678192 5557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:12.678428 5557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:12.679083 5557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:12.679988 5557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:12.680064 5557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:12.680140 5557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:12.680192 5557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:12.690140 5557 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:45029
I20260926 20:16:12.690196 5687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:45029 every 8 connection(s)
I20260926 20:16:12.691250 5557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:12.699838 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 5557
I20260926 20:16:12.700017 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:12.700449 32223 external_mini_cluster.cc:1428] Setting key 0bb7691a98e1de7e9fc8d3e347302092
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:12.707738 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:12.693621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:36669 (local address 127.31.119.254:41931)
0926 20:16:12.693779 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:12.693784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:12.694552 (+ 768us) server_negotiation.cc:407] Connection header received
0926 20:16:12.696017 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:12.696020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:12.696078 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:12.696192 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:12.697535 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.698073 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.699290 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.699534 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.699652 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:12.703150 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:12.703154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:12.703156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:12.703181 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:12.704913 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.705803 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.705806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.705808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.705957 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.706522 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.706533 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.706535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.706739 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:12.706850 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:12.707427 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 20:16:12.707573 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:16:12.708690 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:12.693871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41931 (local address 127.31.119.194:36669)
0926 20:16:12.694350 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:12.694397 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:12.695753 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:12.696377 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:12.696384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:12.696771 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:12.697379 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:12.697395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.698207 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.698211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:12.699153 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:12.699162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.699707 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.699711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:12.699888 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:12.700835 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:12.700852 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:12.702921 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:12.705085 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:12.705092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:12.705133 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:12.705633 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:12.706104 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:12.706107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:12.706109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:12.706394 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:12.706864 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:12.706872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:12.707167 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 20:16:12.707370 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 20:16:12.707745 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":134,"threads_started":1}
I20260926 20:16:12.709971 5688 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41931
I20260926 20:16:12.710228 5688 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
I20260926 20:16:12.710819 5688 heartbeater.cc:511] Master 127.31.119.254:41931 requested a full tablet report, sending...
I20260926 20:16:12.711954 5361 ts_manager.cc:194] Registered new tserver with Master: 78c43f3e18dc4201b865efb76acf4b7b (127.31.119.194:45029)
I20260926 20:16:12.712507 5361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:36669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:16:12.732187 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41931
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:12.850270 5693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:12.850572 5693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:12.850617 5693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:12.854911 5693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:12.854990 5693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:12.855051 5693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:12.860630 5693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:12.861809 5693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:12.862898 5693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:12.872972 5698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:12.873071 5699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:12.873071 5701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:12.873241 5693 server_base.cc:1034] running on GCE node
I20260926 20:16:12.873802 5693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:12.874531 5693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:12.875737 5693 hybrid_clock.cc:648] HybridClock initialized: now 1790453772875710 us; error 53 us; skew 500 ppm
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:12.878872 5693 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:16:12.879868 5693 webserver.cc:460] Webserver started at http://127.31.119.195:45297/ using document root <none> and password file <none>
I20260926 20:16:12.880512 5693 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:12.880609 5693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:12.880941 5693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:12.883301 5693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "8905a18dd3914fa3af5484be6ea42161"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "b5f195f5c008bba461f9ac65afddb385"
server_key_iv: "2fe15c918db307ded48a2cce71654fbf"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.883935 5693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "8905a18dd3914fa3af5484be6ea42161"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "b5f195f5c008bba461f9ac65afddb385"
server_key_iv: "2fe15c918db307ded48a2cce71654fbf"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.888403 5693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:16:12.891532 5708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:12.893090 5693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:16:12.893268 5693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "8905a18dd3914fa3af5484be6ea42161"
format_stamp: "Formatted at 2026-09-26 20:16:12 on dist-test-slave-vnbh"
server_key: "b5f195f5c008bba461f9ac65afddb385"
server_key_iv: "2fe15c918db307ded48a2cce71654fbf"
server_key_version: "encryptionkey@0"
I20260926 20:16:12.893491 5693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:12.904757 5693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:12.908985 5693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:12.909237 5693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:12.909981 5693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:12.910857 5693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:12.910929 5693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:12.911026 5693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:12.911068 5693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:12.920971 5693 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:41071
I20260926 20:16:12.921041 5822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:41071 every 8 connection(s)
I20260926 20:16:12.922109 5693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:12.929817 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 5693
I20260926 20:16:12.929960 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:12.930303 32223 external_mini_cluster.cc:1428] Setting key 9fdbbfdfea22918e4bd3864f85f799af
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:12.937991 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:12.924366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:44571 (local address 127.31.119.254:41931)
0926 20:16:12.924519 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:12.924523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:12.925568 (+ 1045us) server_negotiation.cc:407] Connection header received
0926 20:16:12.927033 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:12.927037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:12.927099 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:12.927183 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:12.928534 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.929073 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.930848 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.931170 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.931295 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:12.934256 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:12.934260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:12.934262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:12.934289 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:12.935670 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.936332 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.936335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.936337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.936431 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.936826 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.936830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.936831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.937094 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:12.937195 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:12.937754 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 20:16:12.937865 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:16:12.939044 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:12.924820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41931 (local address 127.31.119.195:44571)
0926 20:16:12.925387 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:12.925447 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:12.926800 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:12.927383 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:12.927391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:12.927830 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:12.928326 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:12.928340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.929191 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.929194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:12.930711 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:12.930721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.931379 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.931384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:12.931596 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:12.932273 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:12.932291 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:12.934081 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:12.935876 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:12.935882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:12.935896 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:12.936219 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:12.936522 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:12.936531 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:12.936533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:12.936690 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:12.937210 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:12.937216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:12.937499 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 20:16:12.937671 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 20:16:12.937950 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":139,"threads_started":1}
I20260926 20:16:12.940357 5823 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41931
I20260926 20:16:12.940701 5823 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:12.941252 5823 heartbeater.cc:511] Master 127.31.119.254:41931 requested a full tablet report, sending...
I20260926 20:16:12.942535 5361 ts_manager.cc:194] Registered new tserver with Master: 8905a18dd3914fa3af5484be6ea42161 (127.31.119.195:41071)
I20260926 20:16:12.943287 5361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:44571
I20260926 20:16:12.945218 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/test-admin.keytab.
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:12.963913 32223 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 20:16:12.967531 32223 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5cc' of type FILE
W20260926 20:16:12.967514 5830 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:12.965708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.31.119.193:36515)
0926 20:16:12.966033 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:12.966052 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:12.966085 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:16:12.966159 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:12.966164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:12.966361 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:12.966491 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:12.967305 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34474: BlockingRecv error: recv got EOF from 127.0.0.1:34474 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":148,"threads_started":1}
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:12.968776 32223 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:12.970099 32223 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:16:12.981379 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:12.971254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.31.119.193:36515)
0926 20:16:12.971413 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:12.971416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:12.971542 (+ 126us) server_negotiation.cc:407] Connection header received
0926 20:16:12.971667 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:12.971669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:12.971712 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:12.971785 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:12.972350 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.973122 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.974207 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.974642 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.974735 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:12.977354 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:12.977368 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:12.977384 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:12.977415 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:12.979233 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.979809 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.979816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.979817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.979909 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.980190 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.980192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.980194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.980471 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:12.980664 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:12.980920 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 20:16:12.981153 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 20:16:12.985726 5830 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:12.984543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.31.119.193:36515)
0926 20:16:12.984672 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:12.984675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:12.984689 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:16:12.984813 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:12.984815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:12.984851 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:12.984916 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:12.985613 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34498: BlockingRecv error: recv got EOF from 127.0.0.1:34498 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 20:16:12.985847 32223 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:12.986968 32223 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:12.987874 32223 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:16:12.998278 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:12.988811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.31.119.193:36515)
0926 20:16:12.988987 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:12.988990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:12.989005 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:12.989186 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:12.989189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:12.989238 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:12.989408 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:12.990068 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.990655 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.991670 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:12.991904 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:12.992026 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:12.994811 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:12.994814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:12.994816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:12.994843 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:12.996324 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.996855 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.996858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.996859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.996944 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:12.997255 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:12.997258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:12.997259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:12.997560 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:12.997713 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:12.997949 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 20:16:12.998058 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 20:16:13.001494 5830 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:13.000235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.31.119.193:36515)
0926 20:16:13.000403 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:13.000408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:13.000534 (+ 126us) server_negotiation.cc:407] Connection header received
0926 20:16:13.000752 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:13.000754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:13.000793 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:13.000881 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:13.001375 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34514: BlockingRecv error: recv got EOF from 127.0.0.1:34514 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 20:16:13.001678 32223 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453737971229-32223-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:13.002790 32223 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:16:12 dist-test-slave-vnbh krb5kdc[5316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:13.003737 32223 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:16:13 dist-test-slave-vnbh krb5kdc[5316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:16:13.013775 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:13.004810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.31.119.193:36515)
0926 20:16:13.004947 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:13.004950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:13.004979 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:16:13.005148 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:13.005151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:13.005195 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:13.005349 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:13.006013 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:13.006513 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:13.007343 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:13.007578 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:13.007675 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:13.010443 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:13.010446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:13.010448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:13.010476 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:13.012012 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:13.012437 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:13.012440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:13.012441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:13.012530 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:13.012838 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:13.012840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:13.012842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:13.013026 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:13.013169 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:13.013466 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 20:16:13.013650 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:16:13.014909 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 5423
I20260926 20:16:13.021109 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 5557
I20260926 20:16:13.027091 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 5693
I20260926 20:16:13.032676 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 5329
2026-09-26T20:16:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2278 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:16:13 dist-test-slave-vnbh krb5kdc[5843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:16:13 dist-test-slave-vnbh krb5kdc[5843](info): set up 2 sockets
Sep 26 20:16:13 dist-test-slave-vnbh krb5kdc[5843](info): commencing operation
krb5kdc: starting...
W20260926 20:16:14.126709 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:14 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.254.keytab.
Entry for principal oryx/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.254.keytab.
I20260926 20:16:14.265744 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:46179
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.254.keytab
--principal=oryx/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:46179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:14.385347 5856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:14.385735 5856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:14.385825 5856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:14.390079 5856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:14.390178 5856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:14.390214 5856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:14.390254 5856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:14.390295 5856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:14.396121 5856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:46179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.254.keytab
--principal=oryx/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:46179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:14.397434 5856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:14.398559 5856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:14.408098 5861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:14.408119 5864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:14.408272 5856 server_base.cc:1034] running on GCE node
W20260926 20:16:14.408102 5862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:14.409179 5856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:14.410512 5856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:14.411738 5856 hybrid_clock.cc:648] HybridClock initialized: now 1790453774411715 us; error 33 us; skew 500 ppm
Sep 26 20:16:14 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:14.415143 5856 init.cc:377] Logged in from keytab as oryx/127.31.119.254@KRBTEST.COM (short username oryx)
I20260926 20:16:14.416191 5856 webserver.cc:460] Webserver started at http://127.31.119.254:35227/ using document root <none> and password file <none>
I20260926 20:16:14.416795 5856 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:14.416890 5856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:14.417137 5856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:14.419387 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "797fee1bbcb9413db379bc44ca2f43ec"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "6bac461225aeb936ed6290dbfc25c2aa"
server_key_iv: "2d8a72161e37a6cc6403074369a9b94f"
server_key_version: "encryptionkey@0"
I20260926 20:16:14.420012 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "797fee1bbcb9413db379bc44ca2f43ec"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "6bac461225aeb936ed6290dbfc25c2aa"
server_key_iv: "2d8a72161e37a6cc6403074369a9b94f"
server_key_version: "encryptionkey@0"
I20260926 20:16:14.424387 5856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:16:14.427258 5872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:14.428629 5856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:14.428787 5856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "797fee1bbcb9413db379bc44ca2f43ec"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "6bac461225aeb936ed6290dbfc25c2aa"
server_key_iv: "2d8a72161e37a6cc6403074369a9b94f"
server_key_version: "encryptionkey@0"
I20260926 20:16:14.428911 5856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:14.441442 5856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:14.445091 5856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:14.445362 5856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:14.452632 5856 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:46179
I20260926 20:16:14.453014 5924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:46179 every 8 connection(s)
I20260926 20:16:14.453780 5856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:14.457005 5925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:14.463104 5925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec: Bootstrap starting.
I20260926 20:16:14.463256 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 5856
I20260926 20:16:14.463418 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:14.463726 32223 external_mini_cluster.cc:1428] Setting key 41866c380f84931cc748baf1d60fe880
I20260926 20:16:14.466105 5925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:14.467037 5925 log.cc:824] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:14.469121 5925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec: No bootstrap required, opened a new log
I20260926 20:16:14.472688 5925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } }
I20260926 20:16:14.472962 5925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:14.473058 5925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797fee1bbcb9413db379bc44ca2f43ec, State: Initialized, Role: FOLLOWER
Sep 26 20:16:14 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:14.473639 5925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } }
I20260926 20:16:14.473816 5925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:14.473901 5925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:14.474015 5925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:14.474987 5925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } }
I20260926 20:16:14.475414 5925 leader_election.cc:302] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 797fee1bbcb9413db379bc44ca2f43ec; no voters:
I20260926 20:16:14.475829 5925 leader_election.cc:288] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:14.475983 5930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:14.476261 5930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 1 LEADER]: Becoming Leader. State: Replica: 797fee1bbcb9413db379bc44ca2f43ec, State: Running, Role: LEADER
I20260926 20:16:14.476593 5930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } }
I20260926 20:16:14.477367 5925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:14.478511 5931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } }
I20260926 20:16:14.478646 5931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:14.478971 5932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 797fee1bbcb9413db379bc44ca2f43ec. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } }
I20260926 20:16:14.479064 5932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:14.479368 5939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:14.480938 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:14.465188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.31.119.254:46179)
0926 20:16:14.465772 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:14.465791 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:14.465834 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:16:14.466739 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:14.466764 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:14.467226 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:14.467627 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:14.468475 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:14.469587 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:14.470450 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:14.470939 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:14.471121 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:14.473914 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:14.473922 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:14.473941 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:14.474040 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:14.476349 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:14.476921 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:14.476927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:14.476934 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:14.477088 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:14.477517 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:14.477521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:14.477523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:14.478169 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:14.479367 (+ 1198us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:14.479594 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 20:16:14.479930 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":211,"threads_started":1}
I20260926 20:16:14.482529 5939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:14.488561 5939 catalog_manager.cc:1347] Generated new cluster ID: 14dc690dc703444da8ff268d5ac1b4de
I20260926 20:16:14.488682 5939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:14.503561 5939 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:14.504676 5939 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:14.511472 5939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec: Generated new TSK 0
I20260926 20:16:14.512296 5939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab.
Entry for principal oryx/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab.
I20260926 20:16:14.545142 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:46179
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab
--principal=oryx/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:14.690882 5951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:14.691205 5951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:14.691278 5951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:14.695742 5951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:14.695847 5951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:14.695943 5951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:14.702008 5951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab
--principal=oryx/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:46179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:14.703183 5951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:14.704331 5951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:14.714555 5957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:14.714493 5959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:14.714493 5956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:14.714982 5951 server_base.cc:1034] running on GCE node
I20260926 20:16:14.715440 5951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:14.716106 5951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:14.717275 5951 hybrid_clock.cc:648] HybridClock initialized: now 1790453774717258 us; error 30 us; skew 500 ppm
Sep 26 20:16:14 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:14.720664 5951 init.cc:377] Logged in from keytab as oryx/127.31.119.193@KRBTEST.COM (short username oryx)
I20260926 20:16:14.721658 5951 webserver.cc:460] Webserver started at http://127.31.119.193:34847/ using document root <none> and password file <none>
I20260926 20:16:14.722270 5951 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:14.722333 5951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:14.722582 5951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:14.724812 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "7d4e11f6773e4724995df855fed7f7d4"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "7da887ad69c839183ee1107a87f5b3d0"
server_key_iv: "cd8bbcc1259ac4acc2c174278872155d"
server_key_version: "encryptionkey@0"
I20260926 20:16:14.725507 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "7d4e11f6773e4724995df855fed7f7d4"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "7da887ad69c839183ee1107a87f5b3d0"
server_key_iv: "cd8bbcc1259ac4acc2c174278872155d"
server_key_version: "encryptionkey@0"
I20260926 20:16:14.729434 5951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:16:14.732069 5966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:14.733525 5951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:14.733660 5951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "7d4e11f6773e4724995df855fed7f7d4"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "7da887ad69c839183ee1107a87f5b3d0"
server_key_iv: "cd8bbcc1259ac4acc2c174278872155d"
server_key_version: "encryptionkey@0"
I20260926 20:16:14.733783 5951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:14.749413 5951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:14.752738 5951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:14.752957 5951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:14.753646 5951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:14.754561 5951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:14.754640 5951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:14.754734 5951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:14.754786 5951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:14.764264 5951 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:35571
I20260926 20:16:14.764302 6080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:35571 every 8 connection(s)
I20260926 20:16:14.765252 5951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:14.775511 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 5951
I20260926 20:16:14.775671 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
Sep 26 20:16:14 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:14.776104 32223 external_mini_cluster.cc:1428] Setting key 5782ad8743e2133214cb3a50addf99fa
I20260926 20:16:14.781567 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:14.767286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:56457 (local address 127.31.119.254:46179)
0926 20:16:14.767430 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:14.767433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:14.768242 (+ 809us) server_negotiation.cc:407] Connection header received
0926 20:16:14.769958 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:14.769962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:14.770027 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:14.770134 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:14.771406 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:14.771954 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:14.773226 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:14.773557 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:14.773681 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:14.776784 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:14.776788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:14.776792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:14.776825 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:14.778911 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:14.779753 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:14.779757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:14.779759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:14.779907 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:14.780314 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:14.780318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:14.780320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:14.780529 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:14.780648 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:14.781218 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 20:16:14.781380 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:16:14.782488 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:14.767600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.193:56457)
0926 20:16:14.768069 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:14.768116 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:14.769698 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:14.770331 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:14.770338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:14.770735 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:14.771243 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:14.771257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:14.772129 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:14.772131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:14.773072 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:14.773083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:14.773732 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:14.773737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:14.773930 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:14.774595 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:14.774612 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:14.776459 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:14.779082 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:14.779089 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:14.779110 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:14.779598 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:14.780013 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:14.780031 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:14.780033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:14.780207 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:14.780658 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:14.780669 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:14.780963 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 20:16:14.781181 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 20:16:14.781586 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260926 20:16:14.784128 6081 heartbeater.cc:347] Connected to a master server at 127.31.119.254:46179
I20260926 20:16:14.784402 6081 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:14.784973 6081 heartbeater.cc:511] Master 127.31.119.254:46179 requested a full tablet report, sending...
I20260926 20:16:14.786768 5889 ts_manager.cc:194] Registered new tserver with Master: 7d4e11f6773e4724995df855fed7f7d4 (127.31.119.193:35571)
WARNING: no policy specified for oryx/127.31.119.194@KRBTEST.COM; defaulting to no policy
I20260926 20:16:14.787964 5889 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:56457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab.
Entry for principal oryx/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab.
I20260926 20:16:14.808570 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:46179
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab
--principal=oryx/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:14.924063 6086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:14.924374 6086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:14.924461 6086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:14.928771 6086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:14.928877 6086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:14.928978 6086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:14.934646 6086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab
--principal=oryx/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:46179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:14.935837 6086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:14.936934 6086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:14.947161 6094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:14.947156 6091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:14.947196 6092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:14.947422 6086 server_base.cc:1034] running on GCE node
I20260926 20:16:14.947978 6086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:14.948648 6086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:14.949831 6086 hybrid_clock.cc:648] HybridClock initialized: now 1790453774949803 us; error 31 us; skew 500 ppm
Sep 26 20:16:14 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:14.952656 6086 init.cc:377] Logged in from keytab as oryx/127.31.119.194@KRBTEST.COM (short username oryx)
I20260926 20:16:14.953657 6086 webserver.cc:460] Webserver started at http://127.31.119.194:42859/ using document root <none> and password file <none>
I20260926 20:16:14.954267 6086 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:14.954329 6086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:14.954571 6086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:14.956734 6086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "9c0866b3d3c243a1b9349584dbbc8355"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "af47d5da2dfb30db36a5d801026c7268"
server_key_iv: "1db5ab0fded5c0f077935f2329504f4d"
server_key_version: "encryptionkey@0"
I20260926 20:16:14.957404 6086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "9c0866b3d3c243a1b9349584dbbc8355"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "af47d5da2dfb30db36a5d801026c7268"
server_key_iv: "1db5ab0fded5c0f077935f2329504f4d"
server_key_version: "encryptionkey@0"
I20260926 20:16:14.961382 6086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:16:14.964210 6101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:14.965716 6086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:16:14.965857 6086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "9c0866b3d3c243a1b9349584dbbc8355"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "af47d5da2dfb30db36a5d801026c7268"
server_key_iv: "1db5ab0fded5c0f077935f2329504f4d"
server_key_version: "encryptionkey@0"
I20260926 20:16:14.965975 6086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:14.981072 6086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:14.984267 6086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:14.984485 6086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:14.985095 6086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:14.985997 6086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:14.986043 6086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:14.986083 6086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:14.986099 6086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:14.995577 6086 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:42921
I20260926 20:16:14.995684 6215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:42921 every 8 connection(s)
I20260926 20:16:14.996629 6086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:15.005677 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 6086
I20260926 20:16:15.005812 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:15.006116 32223 external_mini_cluster.cc:1428] Setting key 856dfff007d11af11c8ff22b28465842
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:15.013052 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:14.998801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:48465 (local address 127.31.119.254:46179)
0926 20:16:14.998968 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:14.998971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:14.999856 (+ 885us) server_negotiation.cc:407] Connection header received
0926 20:16:15.001216 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.001219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.001267 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.001410 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.002694 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.003248 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.004618 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.004844 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.004943 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.008627 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.008630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.008632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.008659 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.010389 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.011043 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.011050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.011055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.011185 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.011608 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.011614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.011619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.011911 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.012035 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.012599 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.012759 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:16:15.014037 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:14.999164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.194:48465)
0926 20:16:14.999690 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:14.999736 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.000996 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.001643 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.001650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.002035 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.002544 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.002557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.003409 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.003412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.004409 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.004419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.005029 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.005033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.005203 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.006095 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.006113 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.008378 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.010510 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.010517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.010536 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.010912 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.011290 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.011292 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.011294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.011441 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.012059 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.012065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.012326 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.012496 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.012821 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
I20260926 20:16:15.015348 6216 heartbeater.cc:347] Connected to a master server at 127.31.119.254:46179
I20260926 20:16:15.015663 6216 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:15.016317 6216 heartbeater.cc:511] Master 127.31.119.254:46179 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.31.119.195@KRBTEST.COM; defaulting to no policy
I20260926 20:16:15.017726 5889 ts_manager.cc:194] Registered new tserver with Master: 9c0866b3d3c243a1b9349584dbbc8355 (127.31.119.194:42921)
I20260926 20:16:15.018515 5889 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:48465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab.
Entry for principal oryx/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab.
I20260926 20:16:15.037194 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:46179
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab
--principal=oryx/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:15.152478 6221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:15.152838 6221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:15.152890 6221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:15.157051 6221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:15.157124 6221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:15.157188 6221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:15.162740 6221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab
--principal=oryx/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:46179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:15.163964 6221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:15.164978 6221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:15.175603 6229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:15.175774 6226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:15.175705 6221 server_base.cc:1034] running on GCE node
W20260926 20:16:15.175627 6227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:15.176347 6221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:15.177099 6221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:15.178297 6221 hybrid_clock.cc:648] HybridClock initialized: now 1790453775178291 us; error 50 us; skew 500 ppm
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:15.181447 6221 init.cc:377] Logged in from keytab as oryx/127.31.119.195@KRBTEST.COM (short username oryx)
I20260926 20:16:15.182492 6221 webserver.cc:460] Webserver started at http://127.31.119.195:38291/ using document root <none> and password file <none>
I20260926 20:16:15.183189 6221 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:15.183284 6221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:15.183468 6221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:15.185709 6221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "4dea1409891644e685da300de2183ef3"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "9742d84ab463669df28a1882142f8d41"
server_key_iv: "79a2b95397412c4ab43aa3101a746b4a"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.186280 6221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "4dea1409891644e685da300de2183ef3"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "9742d84ab463669df28a1882142f8d41"
server_key_iv: "79a2b95397412c4ab43aa3101a746b4a"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.190348 6221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:15.193042 6236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:15.194635 6221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:15.194752 6221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "4dea1409891644e685da300de2183ef3"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "9742d84ab463669df28a1882142f8d41"
server_key_iv: "79a2b95397412c4ab43aa3101a746b4a"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.194864 6221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:15.213537 6221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:15.217126 6221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:15.217393 6221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:15.217976 6221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:15.218798 6221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:15.218845 6221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:15.218888 6221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:15.218948 6221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:15.228736 6221 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:33481
I20260926 20:16:15.228776 6350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:33481 every 8 connection(s)
I20260926 20:16:15.229789 6221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:15.234273 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 6221
I20260926 20:16:15.234376 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:15.234696 32223 external_mini_cluster.cc:1428] Setting key bd68f2609e494cb7d8a032a83e05a76b
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:15.245383 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.231818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:58179 (local address 127.31.119.254:46179)
0926 20:16:15.231966 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.231971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.233037 (+ 1066us) server_negotiation.cc:407] Connection header received
0926 20:16:15.234587 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.234591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.234651 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.234739 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.236403 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.237066 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.238538 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.238766 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.238864 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.241657 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.241664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.241666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.241696 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.243325 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.243872 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.243875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.243877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.243956 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.244267 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.244270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.244271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.244449 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.244521 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.245080 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.245192 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:16:15.246270 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.232268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.195:58179)
0926 20:16:15.232845 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.232908 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.234293 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.234953 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.234960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.235530 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.236252 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.236273 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.237214 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.237220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.238395 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.238405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.238913 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.238916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.239096 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.239716 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.239734 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.241417 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.243453 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.243458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.243474 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.243770 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.244036 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.244038 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.244040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.244181 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.244577 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.244582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.244837 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.245008 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.245345 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":106,"threads_started":1}
I20260926 20:16:15.247656 6351 heartbeater.cc:347] Connected to a master server at 127.31.119.254:46179
I20260926 20:16:15.247936 6351 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:15.248566 6351 heartbeater.cc:511] Master 127.31.119.254:46179 requested a full tablet report, sending...
I20260926 20:16:15.249698 5889 ts_manager.cc:194] Registered new tserver with Master: 4dea1409891644e685da300de2183ef3 (127.31.119.195:33481)
I20260926 20:16:15.250293 5889 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:58179
I20260926 20:16:15.261782 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.119.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.253.keytab.
Entry for principal oryx/127.31.119.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.253.keytab.
I20260926 20:16:15.292222 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.253:34189
--webserver_interface=127.31.119.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.253.keytab
--principal=oryx/127.31.119.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:46179,127.31.119.253:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:15.408751 6356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:15.409106 6356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:15.409181 6356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:15.413415 6356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:15.413533 6356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:15.413568 6356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:15.413606 6356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:15.413662 6356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:15.419343 6356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:46179,127.31.119.253:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.253.keytab
--principal=oryx/127.31.119.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.253:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.119.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
W20260926 20:16:15.419669 6356 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.119.254:46179,127.31.119.253:34189'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 20:16:15.420646 6356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:15.421742 6356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:15.430678 6361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:15.430687 6362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:15.430687 6364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:15.431147 6356 server_base.cc:1034] running on GCE node
I20260926 20:16:15.431733 6356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:15.433043 6356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:15.434283 6356 hybrid_clock.cc:648] HybridClock initialized: now 1790453775434234 us; error 63 us; skew 500 ppm
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:15.437443 6356 init.cc:377] Logged in from keytab as oryx/127.31.119.253@KRBTEST.COM (short username oryx)
I20260926 20:16:15.438619 6356 webserver.cc:460] Webserver started at http://127.31.119.253:35891/ using document root <none> and password file <none>
I20260926 20:16:15.439312 6356 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:15.439422 6356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:15.439718 6356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:15.442050 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data/instance:
uuid: "c5f033f13aa84455b5f010f47419e62a"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "4087b8eef728410daca32d672fa68bfa"
server_key_iv: "151ce2c4e2bf127edcd5503572beb212"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.442662 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal/instance:
uuid: "c5f033f13aa84455b5f010f47419e62a"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "4087b8eef728410daca32d672fa68bfa"
server_key_iv: "151ce2c4e2bf127edcd5503572beb212"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.446555 6356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:15.449196 6371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:15.450569 6356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:15.450708 6356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal
uuid: "c5f033f13aa84455b5f010f47419e62a"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "4087b8eef728410daca32d672fa68bfa"
server_key_iv: "151ce2c4e2bf127edcd5503572beb212"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.450821 6356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:15.467892 6356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:15.471483 6356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:15.471695 6356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:15.479092 6356 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.253:34189
I20260926 20:16:15.479337 6424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.253:34189 every 8 connection(s)
I20260926 20:16:15.480329 6356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
I20260926 20:16:15.483598 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:15.487784 6425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:16:15.489557 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 6356
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.253@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.119.253@KRBTEST.COM
I20260926 20:16:15.502934 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.490036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45210 (local address 127.31.119.254:46179)
0926 20:16:15.490220 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.490226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.491192 (+ 966us) server_negotiation.cc:407] Connection header received
0926 20:16:15.492162 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.492165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.492220 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.492325 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.493331 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.493873 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.494950 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.495178 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.495290 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.498336 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.498343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.498349 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.498390 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.500161 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.500893 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.500896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.500898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.500981 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.501413 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.501419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.501423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.501756 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.501868 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.502571 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.502730 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:16:15.504161 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.490526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:45210)
0926 20:16:15.491022 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.491073 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.491951 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.492371 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.492377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.492753 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.493157 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.493171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.494037 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.494039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.494809 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.494817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.495334 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.495338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.495549 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.496261 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.496283 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.498129 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.500298 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.500309 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.500324 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.500778 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.501107 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.501114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.501118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.501332 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.501879 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.501887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.502266 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.502507 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.502946 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":44,"thread_start_us":141,"threads_started":1}
I20260926 20:16:15.504161 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.490526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46730 (local address 127.31.119.253:34189)
0926 20:16:15.491399 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.491408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.491447 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:16:15.491982 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.492008 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.492213 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.492428 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.493277 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.494233 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.495083 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.495515 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.495638 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.499173 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.499179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.499192 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.499214 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.501213 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.501879 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.501885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.501887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.501973 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.502250 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.502253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.502254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.502539 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.502738 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.502962 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.503283 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":737,"spinlock_wait_cycles":29312,"thread_start_us":192,"threads_started":1}
I20260926 20:16:15.505722 6425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.253@KRBTEST.COM for oryx/127.31.119.253@KRBTEST.COM
I20260926 20:16:15.513686 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.506236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:34189 (local address 127.0.0.1:46736)
0926 20:16:15.506341 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.506355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.506429 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.506693 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.506696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.506836 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.506929 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.506935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.507744 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.507747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.508268 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.508274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.508730 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.508732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.508802 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.509242 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.509252 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.510567 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.512333 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.512335 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.512337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.512520 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.512796 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.512798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.512800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.512869 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.513194 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.513196 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.513269 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.513399 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.513510 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:16:15.513700 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.506299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.31.119.253:34189)
0926 20:16:15.506403 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.506408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.506423 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:15.506547 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.506549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.506588 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.506682 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.507097 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.507629 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.508379 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.508634 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.508717 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.510711 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.510714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.510716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.510736 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.512240 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.512626 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.512629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.512631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.512704 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.512945 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.512948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.512949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.513119 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.513193 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.513446 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.513570 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":37}
I20260926 20:16:15.516938 6425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: Bootstrap starting.
I20260926 20:16:15.519280 6425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:15.520094 6425 log.cc:824] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:15.522017 6425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: No bootstrap required, opened a new log
I20260926 20:16:15.525209 6425 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } }
I20260926 20:16:15.525520 6425 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:15.525575 6425 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f033f13aa84455b5f010f47419e62a, State: Initialized, Role: FOLLOWER
I20260926 20:16:15.525967 6425 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } }
I20260926 20:16:15.526413 6431 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } } }
I20260926 20:16:15.526535 6431 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:16:15.527238 6425 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:16:15.531864 6442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:16:15.531960 6442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:16:15.534376 6356 master_runner.cc:428] Detected that this master c5f033f13aa84455b5f010f47419e62a is joining an existing cluster
I20260926 20:16:15.534453 6356 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.119.253:34189
I20260926 20:16:15.545420 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.536519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.31.119.254:46179)
0926 20:16:15.536668 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.536689 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.536843 (+ 154us) server_negotiation.cc:407] Connection header received
0926 20:16:15.537049 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.537052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.537107 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.537151 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.537839 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.538686 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.539594 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.539960 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.540084 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.541277 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.541299 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.541302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.541332 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.543207 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.543986 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.544003 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.544005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.544104 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.544445 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.544449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.544451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.544697 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.544789 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.545094 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.545214 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:16:15.545434 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.536431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:45218)
0926 20:16:15.536756 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.536769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.536917 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.537269 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.537271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.537512 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.537695 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.537701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.538794 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.538796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.539473 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.539481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.540089 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.540093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.540178 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.540759 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.540773 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.541150 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.543336 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.543342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.543347 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.543606 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.544216 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.544219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.544221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.544332 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.544834 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.544836 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.544912 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.545027 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.545180 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":99,"threads_started":1}
I20260926 20:16:15.546383 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.536892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.31.119.253:34189)
0926 20:16:15.536998 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.537004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.537582 (+ 578us) server_negotiation.cc:407] Connection header received
0926 20:16:15.537692 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.537695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.537738 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.537832 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.539339 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.539870 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.540950 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.541319 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.541425 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.542804 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.542807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.542810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.542840 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.544646 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.545107 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.545110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.545112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.545177 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.545482 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.545485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.545487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.545663 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.545727 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.546199 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.546283 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:16:15.548064 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.536851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:34189 (local address 127.0.0.1:46742)
0926 20:16:15.537483 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.537497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.537606 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.538841 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.538844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.539095 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.539211 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.539218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.539966 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.539972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.540821 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.540835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.541428 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.541432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.541553 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.542157 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.542176 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.542643 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.544751 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.544754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.544756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.544994 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.545271 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.545274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.545276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.545378 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.545914 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.545917 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.545997 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.546121 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.547872 (+ 1751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"spinlock_wait_cycles":3328,"thread_start_us":79,"threads_started":1}
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.254@KRBTEST.COM for oryx/127.31.119.253@KRBTEST.COM
I20260926 20:16:15.561331 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.551012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.31.119.253:34189)
0926 20:16:15.551170 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.551174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.551586 (+ 412us) server_negotiation.cc:407] Connection header received
0926 20:16:15.551782 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.551784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.551828 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.551895 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.552721 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.553201 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.554556 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.554763 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.554906 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.557551 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.557555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.557558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.557583 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.559103 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.559605 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.559610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.559612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.559731 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.560225 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.560229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.560231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.560457 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.560559 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.561003 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.561125 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:16:15.561352 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.550990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:34189 (local address 127.0.0.1:46744)
0926 20:16:15.551437 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.551496 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.551639 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.551952 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.551958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.552432 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.552583 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.552592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.553329 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.553332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.554427 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.554438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.554915 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.554919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.555043 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.555632 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.555648 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.557419 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.559225 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.559237 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.559244 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.559492 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.559857 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.559862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.559866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.559973 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.560578 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.560586 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.560759 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.560939 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.561143 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":140,"threads_started":1}
I20260926 20:16:15.562877 5889 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.119.253:34189: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c5f033f13aa84455b5f010f47419e62a"
member_type: NON_VOTER
last_known_addr {
host: "127.31.119.253"
port: 34189
}
attrs {
promote: true
}
}
dest_uuid: "797fee1bbcb9413db379bc44ca2f43ec"
cas_config_opid_index: -1
I20260926 20:16:15.563565 5889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } }
I20260926 20:16:15.565474 6454 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c5f033f13aa84455b5f010f47419e62a (127.31.119.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } } }
I20260926 20:16:15.566392 6398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:15.567462 6454 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.119.253:34189
WARNING: no policy specified for oryx/127.31.119.252@KRBTEST.COM; defaulting to no policy
I20260926 20:16:15.567833 6454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } } }
I20260926 20:16:15.567924 6454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:15.567894 6398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 FOLLOWER]: Refusing update from remote peer 797fee1bbcb9413db379bc44ca2f43ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 20:16:15.567904 6356 master.cc:537] Master@127.31.119.253:34189 shutting down...
I20260926 20:16:15.568655 6455 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.252@KRBTEST.COM" created.
W20260926 20:16:15.573028 5875 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 20:16:15.573541 6356 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:15.573856 6356 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:15.573990 6356 tablet_replica.cc:330] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: stopping tablet replica
W20260926 20:16:15.573999 5875 consensus_peers.cc:487] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec -> Peer c5f033f13aa84455b5f010f47419e62a (127.31.119.253:34189): Couldn't send request to peer c5f033f13aa84455b5f010f47419e62a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:16:15.589488 6356 master.cc:552] Master@127.31.119.253:34189 shutdown complete.
I20260926 20:16:15.589555 6356 master_runner.cc:305] Clearing existing system tablet
I20260926 20:16:15.589808 6356 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:15.589861 6356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:15.592234 6356 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:16:15.593190 6460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:15.593483 6356 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:15.593636 6356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:15.593678 6356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal
uuid: "c5f033f13aa84455b5f010f47419e62a"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "4087b8eef728410daca32d672fa68bfa"
server_key_iv: "151ce2c4e2bf127edcd5503572beb212"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.595723 6356 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.252.keytab.
Entry for principal oryx/127.31.119.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.252.keytab.
I20260926 20:16:15.597661 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.252:43255
--webserver_interface=127.31.119.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.252.keytab
--principal=oryx/127.31.119.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:46179,127.31.119.253:34189,127.31.119.252:43255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:16:15.598294 6356 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:16:15.598392 6356 log.cc:1196] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 20:16:15.598673 6356 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: Deleting consensus metadata
I20260926 20:16:15.598927 6356 master_runner.cc:315] Copying system tablet from 127.31.119.254:46179
I20260926 20:16:15.600149 6356 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.119.254:46179
I20260926 20:16:15.608413 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.600824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.31.119.254:46179)
0926 20:16:15.600957 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.600962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.601208 (+ 246us) server_negotiation.cc:407] Connection header received
0926 20:16:15.601447 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.601450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.601508 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.601607 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.602100 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.602647 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.603505 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.603745 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.603860 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.605048 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.605052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.605054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.605081 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.606704 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.607167 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.607171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.607173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.607257 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.607544 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.607548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.607549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.607738 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.607848 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.608132 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.608245 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:16:15.608425 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.600760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:45230)
0926 20:16:15.601108 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.601120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.601257 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.601625 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.601628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.601834 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.601946 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.601952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.602771 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.602776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.603377 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.603386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.603867 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.603870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.603968 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.604548 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.604560 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.604896 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.606837 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.606840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.606842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.607061 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.607357 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.607360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.607362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.607449 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.607836 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.607839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.607958 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.608074 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.608233 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":104,"threads_started":1}
I20260926 20:16:15.609416 5909 tablet_copy_service.cc:140] P 797fee1bbcb9413db379bc44ca2f43ec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c5f033f13aa84455b5f010f47419e62a ({username='oryx', principal='oryx/127.31.119.253@KRBTEST.COM'} at 127.0.0.1:45230)
I20260926 20:16:15.609611 5909 tablet_copy_service.cc:161] P 797fee1bbcb9413db379bc44ca2f43ec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c5f033f13aa84455b5f010f47419e62a at {username='oryx', principal='oryx/127.31.119.253@KRBTEST.COM'} at 127.0.0.1:45230: session id = c5f033f13aa84455b5f010f47419e62a-00000000000000000000000000000000
I20260926 20:16:15.611143 5909 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:16:15.612735 6356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:16:15.614792 6356 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: tablet copy: Starting download of 0 data blocks...
I20260926 20:16:15.615036 6356 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: tablet copy: Starting download of 1 WAL segments...
I20260926 20:16:15.616710 6356 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:16:15.618465 5909 tablet_copy_service.cc:342] P 797fee1bbcb9413db379bc44ca2f43ec: Request end of tablet copy session c5f033f13aa84455b5f010f47419e62a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.119.253@KRBTEST.COM'} at 127.0.0.1:45230
I20260926 20:16:15.618600 5909 tablet_copy_service.cc:433] P 797fee1bbcb9413db379bc44ca2f43ec: ending tablet copy session c5f033f13aa84455b5f010f47419e62a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c5f033f13aa84455b5f010f47419e62a
I20260926 20:16:15.626308 6356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:15.628135 6468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:15.628135 6471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:15.628489 6469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:15.629479 6356 server_base.cc:1034] running on GCE node
I20260926 20:16:15.629729 6356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:15.630057 6356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:15.631202 6356 hybrid_clock.cc:648] HybridClock initialized: now 1790453775631202 us; error 50 us; skew 500 ppm
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:15.632931 6356 init.cc:377] Logged in from keytab as oryx/127.31.119.253@KRBTEST.COM (short username oryx)
I20260926 20:16:15.633543 6356 webserver.cc:460] Webserver started at http://127.31.119.253:33343/ using document root <none> and password file <none>
I20260926 20:16:15.633756 6356 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:15.633810 6356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:15.635051 6356 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:15.636061 6478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:15.636292 6356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:15.636368 6356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal
uuid: "c5f033f13aa84455b5f010f47419e62a"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "4087b8eef728410daca32d672fa68bfa"
server_key_iv: "151ce2c4e2bf127edcd5503572beb212"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.636449 6356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:15.665419 6356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:15.667539 6356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:15.672605 6356 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.253:34189
I20260926 20:16:15.672827 6531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.253:34189 every 8 connection(s)
I20260926 20:16:15.673254 6356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
I20260926 20:16:15.678788 6532 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:16:15.679701 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: Bootstrap starting.
I20260926 20:16:15.688043 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:16:15.688382 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: Bootstrap complete.
I20260926 20:16:15.688890 6532 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } }
I20260926 20:16:15.689036 6532 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f033f13aa84455b5f010f47419e62a, State: Initialized, Role: LEARNER
I20260926 20:16:15.689154 6532 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } }
I20260926 20:16:15.689528 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } } }
I20260926 20:16:15.689693 6532 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:15.689699 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: This master's current role is: LEARNER
I20260926 20:16:15.692998 6546 catalog_manager.cc:1259] Loaded cluster ID: 14dc690dc703444da8ff268d5ac1b4de
I20260926 20:16:15.693122 6546 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: loading cluster ID for follower catalog manager: success
I20260926 20:16:15.694981 6546 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: acquiring CA information for follower catalog manager: success
I20260926 20:16:15.696105 6546 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 20:16:15.755650 6461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:15.755939 6461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:15.755982 6461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:15.760236 6461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:15.760308 6461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:15.760329 6461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:15.760345 6461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:15.760361 6461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:15.766126 6461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:46179,127.31.119.253:34189,127.31.119.252:43255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.252.keytab
--principal=oryx/127.31.119.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.252:43255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.119.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:15.767230 6461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:15.768278 6461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:15.777303 6461 server_base.cc:1034] running on GCE node
W20260926 20:16:15.777207 6552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:15.777227 6554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:15.777241 6551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:15.778256 6461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:15.779536 6461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:15.780745 6461 hybrid_clock.cc:648] HybridClock initialized: now 1790453775780701 us; error 49 us; skew 500 ppm
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:15.784451 6461 init.cc:377] Logged in from keytab as oryx/127.31.119.252@KRBTEST.COM (short username oryx)
I20260926 20:16:15.785527 6461 webserver.cc:460] Webserver started at http://127.31.119.252:39049/ using document root <none> and password file <none>
I20260926 20:16:15.786127 6461 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:15.786221 6461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:15.786438 6461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:15.788545 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data/instance:
uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "1ad89dd1dbbd3b38dc375767493001e1"
server_key_iv: "91e1b55f3ae53b5bc2b9eb4cc42c6c48"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.789127 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal/instance:
uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "1ad89dd1dbbd3b38dc375767493001e1"
server_key_iv: "91e1b55f3ae53b5bc2b9eb4cc42c6c48"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.790834 6081 heartbeater.cc:503] Master 127.31.119.254:46179 was elected leader, sending a full tablet report...
I20260926 20:16:15.793388 6461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 20:16:15.796020 6561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:15.797864 6461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:15.798039 6461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal
uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "1ad89dd1dbbd3b38dc375767493001e1"
server_key_iv: "91e1b55f3ae53b5bc2b9eb4cc42c6c48"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.798182 6461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:15.815078 6461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:15.818480 6461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:15.818727 6461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:15.826164 6461 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.252:43255
I20260926 20:16:15.826484 6613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.252:43255 every 8 connection(s)
I20260926 20:16:15.827222 6461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
I20260926 20:16:15.830282 6614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:15.834513 6614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:16:15.836220 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 6461
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.252@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.119.252@KRBTEST.COM
I20260926 20:16:15.848963 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.837076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.31.119.254:46179)
0926 20:16:15.837423 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.837428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.837996 (+ 568us) server_negotiation.cc:407] Connection header received
0926 20:16:15.838834 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.838837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.838897 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.838994 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.839914 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.840463 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.841562 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.841789 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.841904 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.844640 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.844644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.844647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.844677 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.846505 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.847196 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.847199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.847200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.847287 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.847710 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.847715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.847717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.847924 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.848037 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.848616 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.848743 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260926 20:16:15.849983 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.837268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.31.119.252:43255)
0926 20:16:15.837713 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.837729 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.837769 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:16:15.838558 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.838623 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.838858 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.839060 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.839930 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.840774 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.841690 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.842144 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.842266 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.845208 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.845218 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.845235 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.845264 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.847148 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.847613 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.847617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.847619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.847697 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.847981 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.847986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.847988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.848301 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.848513 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.848709 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.848904 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":339,"thread_start_us":151,"threads_started":1}
I20260926 20:16:15.849983 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.837212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:45244)
0926 20:16:15.837867 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.837906 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.838583 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.839048 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.839054 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.839415 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.839769 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.839783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.840602 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.840606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.841433 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.841442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.841942 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.841946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.842185 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.842788 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.842804 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.844479 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.846630 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.846637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.846649 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.847068 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.847438 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.847441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.847445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.847604 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.848044 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.848053 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.848319 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.848572 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.849410 (+ 838us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":579,"spinlock_wait_cycles":65408,"thread_start_us":214,"threads_started":1}
I20260926 20:16:15.851389 6614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.252@KRBTEST.COM for oryx/127.31.119.253@KRBTEST.COM
I20260926 20:16:15.860558 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.851916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:34189 (local address 127.0.0.1:46752)
0926 20:16:15.852357 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.852370 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.852433 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.852687 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.852690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.852830 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.852926 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.852940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.854001 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.854004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.854556 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.854564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.855146 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.855151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.855237 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.855828 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.855840 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.857124 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.859088 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.859090 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.859092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.859284 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.859576 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.859578 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.859579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.859651 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.860057 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.860060 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.860144 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.860238 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.860350 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355}
I20260926 20:16:15.860579 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.852097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.31.119.253:34189)
0926 20:16:15.852397 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.852400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.852413 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:16:15.852514 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.852516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.852562 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.852630 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.853085 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.853873 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.854667 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.854921 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.855004 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.857364 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.857368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.857370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.857396 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.858972 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.859370 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.859372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.859374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.859461 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.859728 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.859730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.859731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.859923 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.860002 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.860288 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.860389 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":105,"threads_started":1}
I20260926 20:16:15.861485 6614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.252@KRBTEST.COM for oryx/127.31.119.252@KRBTEST.COM
I20260926 20:16:15.870090 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.862075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.31.119.252:43255)
0926 20:16:15.862281 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.862287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.862301 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:16:15.862528 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.862531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.862576 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.862660 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.862991 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.863507 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.864251 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.864480 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.864588 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.866848 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.866854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.866861 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.866882 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.868337 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.868845 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.868851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.868853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.868925 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.869210 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.869213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.869215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.869443 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.869551 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.869779 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.869928 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":143}
I20260926 20:16:15.870090 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.861967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43255 (local address 127.0.0.1:52772)
0926 20:16:15.862238 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.862252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.862338 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.862667 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.862670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.862826 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.862887 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.862892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.863621 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.863625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.864136 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.864143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.864585 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.864588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.864717 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.865194 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.865206 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.866611 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.868556 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.868561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.868563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.868752 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.869028 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.869031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.869032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.869109 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.869539 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.869541 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.869621 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.869738 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.869878 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":40}
I20260926 20:16:15.873859 6614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Bootstrap starting.
I20260926 20:16:15.876206 6614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:15.876967 6614 log.cc:824] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:15.879277 6614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: No bootstrap required, opened a new log
I20260926 20:16:15.882339 6614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } }
I20260926 20:16:15.882551 6614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:15.882586 6614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebd8eaf5f5c4d87918f3b07d6c25a20, State: Initialized, Role: FOLLOWER
I20260926 20:16:15.882965 6614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } }
I20260926 20:16:15.883416 6620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } } }
I20260926 20:16:15.883545 6620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:16:15.884243 6614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:16:15.889370 6632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:16:15.889467 6632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:16:15.892840 6461 master_runner.cc:428] Detected that this master 4ebd8eaf5f5c4d87918f3b07d6c25a20 is joining an existing cluster
I20260926 20:16:15.892935 6461 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.119.252:43255
I20260926 20:16:15.903626 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 5951
I20260926 20:16:15.905435 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.895202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43255 (local address 127.0.0.1:52776)
0926 20:16:15.896091 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.896104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.896224 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.896543 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.896545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.896824 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.896930 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.896938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.897876 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.897878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.899231 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.899239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.900184 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.900188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.900301 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.900720 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.900735 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.901214 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.903358 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.903362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.903364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.903616 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.904355 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.904361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.904372 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.904474 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.904920 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.904922 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.905008 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.905092 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.905200 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":4,"thread_start_us":108,"threads_started":1}
I20260926 20:16:15.905473 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.895719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.31.119.252:43255)
0926 20:16:15.895828 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.895832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.896167 (+ 335us) server_negotiation.cc:407] Connection header received
0926 20:16:15.896327 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.896329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.896366 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.896410 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.897046 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.897767 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.899339 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.899651 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.900165 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.901406 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.901412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.901419 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.901442 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.903245 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.903718 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.903725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.903730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.903819 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.904588 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.904592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.904594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.904819 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.904920 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.905163 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.905277 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":43}
I20260926 20:16:15.910190 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:35571
--local_ip_for_outbound_sockets=127.31.119.193
--tserver_master_addrs=127.31.119.254:46179
--webserver_port=34847
--webserver_interface=127.31.119.193
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab
--principal=oryx/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.119.254:46179,127.31.119.253:34189
--tserver_master_addrs=127.31.119.254:46179,127.31.119.253:34189,127.31.119.252:43255 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:16:15.918720 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.895682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.31.119.253:34189)
0926 20:16:15.895802 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.895807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.896188 (+ 381us) server_negotiation.cc:407] Connection header received
0926 20:16:15.896372 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.896375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.896427 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.896482 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.897223 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.898058 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.899266 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.899629 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.899709 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.914817 (+ 15108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.914822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.914825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.914859 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.916668 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.917194 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.917198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.917200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.917275 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.917664 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.917667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.917669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.917896 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.917970 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.918401 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.918551 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:16:15.919334 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.895809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:34189 (local address 127.0.0.1:46768)
0926 20:16:15.896101 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.896115 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.896259 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.896626 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.896629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.896937 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.897074 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.897082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.898210 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.898213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.899128 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.899138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.899859 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.899863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.899953 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.913967 (+ 14014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.913986 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.914589 (+ 603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.916801 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.916804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.916806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.917085 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.917425 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.917428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.917430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.917552 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.918047 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.918050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.918220 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.918940 (+ 720us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.919105 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":21,"spinlock_wait_cycles":1408,"thread_start_us":92,"threads_started":1}
I20260926 20:16:15.919633 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.895255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.31.119.254:46179)
0926 20:16:15.895603 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.895608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.895623 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:15.895703 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.895708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.895761 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.895820 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.896546 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.897435 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.898758 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.899136 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.899216 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.914040 (+ 14824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.914048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.914054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.914091 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.916531 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.917164 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.917171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.917175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.917272 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.917664 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.917667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.917669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.917896 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.917982 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.919328 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.919466 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260926 20:16:15.920143 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.895170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:45246)
0926 20:16:15.895485 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.895500 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.895604 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.895996 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.895999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.896247 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.896413 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.896420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.897566 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.897569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.898624 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.898633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.899422 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.899425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.899506 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.900137 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.900153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.913781 (+ 13628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.916686 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.916691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.916695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.917030 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.917425 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.917428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.917430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.917546 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.918047 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.918050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.919123 (+ 1073us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.919792 (+ 669us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.919980 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":55,"spinlock_wait_cycles":5120,"thread_start_us":97,"threads_started":1}
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453774, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.254@KRBTEST.COM for oryx/127.31.119.252@KRBTEST.COM
I20260926 20:16:15.932174 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.923841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43255 (local address 127.0.0.1:52782)
0926 20:16:15.924028 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.924040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.924157 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.924445 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.924448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.924641 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.924731 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.924736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.925574 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.925578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.926286 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.926295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.926769 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.926772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.926868 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.927425 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.927438 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.928706 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.930567 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.930571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.930573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.930789 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.931078 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.931081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.931083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.931174 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.931575 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.931579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.931697 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.931808 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.931986 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 20:16:15.932204 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.923886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.31.119.252:43255)
0926 20:16:15.924003 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.924008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.924109 (+ 101us) server_negotiation.cc:407] Connection header received
0926 20:16:15.924292 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.924295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.924358 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.924429 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.924841 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.925464 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.926409 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.926650 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.926762 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.928847 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.928855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.928858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.928880 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.930400 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.930892 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.930896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.930898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.930982 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.931274 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.931278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.931279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.931465 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.931586 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.931862 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.932038 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:16:15.933616 5889 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.119.252:43255: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20"
member_type: NON_VOTER
last_known_addr {
host: "127.31.119.252"
port: 43255
}
attrs {
promote: true
}
}
dest_uuid: "797fee1bbcb9413db379bc44ca2f43ec"
cas_config_opid_index: 5
I20260926 20:16:15.934242 5889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } }
I20260926 20:16:15.935397 6454 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4ebd8eaf5f5c4d87918f3b07d6c25a20 (127.31.119.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:15.936841 6454 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.119.252:43255
I20260926 20:16:15.937032 6454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:15.937153 6454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:15.938218 6588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:15.940289 6588 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 FOLLOWER]: Refusing update from remote peer 797fee1bbcb9413db379bc44ca2f43ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:16:15.941037 6461 master.cc:537] Master@127.31.119.252:43255 shutting down...
I20260926 20:16:15.942899 6453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:15.943055 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.935633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:34189 (local address 127.0.0.1:46782)
0926 20:16:15.937797 (+ 2164us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.937813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.937897 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.938354 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.938358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.938376 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:15.938498 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.938505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.939683 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.939688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.941031 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.941043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.942552 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.942559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.942690 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.942748 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.942825 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.942917 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2045}
I20260926 20:16:15.943475 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.935964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.31.119.253:34189)
0926 20:16:15.936099 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.936103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.937982 (+ 1879us) server_negotiation.cc:407] Connection header received
0926 20:16:15.938042 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.938045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.938107 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.938206 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:15.938636 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.939549 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.941179 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.942152 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.942288 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.942478 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.943259 (+ 781us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.943346 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:16:15.943964 6505 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 LEARNER]: Refusing update from remote peer 797fee1bbcb9413db379bc44ca2f43ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:16:15.944972 6453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260926 20:16:15.945626 5874 consensus_peers.cc:487] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec -> Peer 4ebd8eaf5f5c4d87918f3b07d6c25a20 (127.31.119.252:43255): Couldn't send request to peer 4ebd8eaf5f5c4d87918f3b07d6c25a20. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:16:15.946851 6505 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4ebd8eaf5f5c4d87918f3b07d6c25a20 (127.31.119.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:15.948906 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 797fee1bbcb9413db379bc44ca2f43ec. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:15.949045 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: This master's current role is: LEARNER
I20260926 20:16:15.949138 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: NON_VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: true } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:15.949215 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: This master's current role is: LEARNER
I20260926 20:16:15.950487 6453 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec: attempting to promote NON_VOTER c5f033f13aa84455b5f010f47419e62a to VOTER
I20260926 20:16:15.950970 6453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } }
I20260926 20:16:15.953431 6461 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:15.953795 6461 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 20:16:15.953847 5874 consensus_peers.cc:487] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec -> Peer 4ebd8eaf5f5c4d87918f3b07d6c25a20 (127.31.119.252:43255): Couldn't send request to peer 4ebd8eaf5f5c4d87918f3b07d6c25a20. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:16:15.954011 6461 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: stopping tablet replica
I20260926 20:16:15.954349 6505 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 LEARNER]: Refusing update from remote peer 797fee1bbcb9413db379bc44ca2f43ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 20:16:15.954798 6454 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 20:16:15.957617 6454 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f033f13aa84455b5f010f47419e62a (127.31.119.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:15.959523 6461 master.cc:552] Master@127.31.119.252:43255 shutdown complete.
I20260926 20:16:15.959630 6461 master_runner.cc:305] Clearing existing system tablet
I20260926 20:16:15.959931 6461 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:15.960043 6461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:15.960224 6504 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f033f13aa84455b5f010f47419e62a (127.31.119.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:15.961519 6454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:15.961723 6454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:15.961971 6454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:15.962046 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:15.962152 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:16:15.962124 6454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:15.962893 6461 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:16:15.963773 6645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:15.963980 6461 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:15.964126 6461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:15.964164 6461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal
uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "1ad89dd1dbbd3b38dc375767493001e1"
server_key_iv: "91e1b55f3ae53b5bc2b9eb4cc42c6c48"
server_key_version: "encryptionkey@0"
I20260926 20:16:15.965919 6461 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:15.967900 6461 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:16:15.968014 6461 log.cc:1196] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 20:16:15.968271 6461 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Deleting consensus metadata
I20260926 20:16:15.968453 6461 master_runner.cc:315] Copying system tablet from 127.31.119.254:46179
I20260926 20:16:15.969682 6461 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.119.254:46179
I20260926 20:16:15.981153 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.970261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:45256)
0926 20:16:15.973539 (+ 3278us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:15.973563 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:15.973720 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:15.974012 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:15.974015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:15.974228 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:15.974314 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.974319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.975144 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.975151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.975770 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:15.975779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.976263 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.976267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:15.976354 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.977002 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:15.977023 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:15.977448 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:15.979472 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.979475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.979477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.979769 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.980060 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:15.980064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:15.980066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:15.980155 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:15.980552 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:15.980556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:15.980663 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 20:16:15.980777 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:16:15.980944 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3190,"thread_start_us":95,"threads_started":1}
I20260926 20:16:15.981729 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:15.970320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.31.119.254:46179)
0926 20:16:15.970455 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:15.970460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:15.973678 (+ 3218us) server_negotiation.cc:407] Connection header received
0926 20:16:15.973817 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:15.973820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:15.973876 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:15.973972 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:15.974451 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.975020 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.975904 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:15.976142 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:15.976259 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:15.977604 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:15.977607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:15.977610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:15.977644 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:15.979300 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.979872 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.979876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.979877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.979968 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:15.980253 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:15.980257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:15.980258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:15.980454 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:15.980564 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:15.980831 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 20:16:15.981075 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:16:15.982185 5909 tablet_copy_service.cc:140] P 797fee1bbcb9413db379bc44ca2f43ec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4ebd8eaf5f5c4d87918f3b07d6c25a20 ({username='oryx', principal='oryx/127.31.119.252@KRBTEST.COM'} at 127.0.0.1:45256)
I20260926 20:16:15.982313 5909 tablet_copy_service.cc:161] P 797fee1bbcb9413db379bc44ca2f43ec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4ebd8eaf5f5c4d87918f3b07d6c25a20 at {username='oryx', principal='oryx/127.31.119.252@KRBTEST.COM'} at 127.0.0.1:45256: session id = 4ebd8eaf5f5c4d87918f3b07d6c25a20-00000000000000000000000000000000
I20260926 20:16:15.983206 5909 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:16:15.984645 6461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:16:15.986966 6461 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: tablet copy: Starting download of 0 data blocks...
I20260926 20:16:15.987178 6461 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: tablet copy: Starting download of 1 WAL segments...
I20260926 20:16:15.988595 6461 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:16:15.990345 5909 tablet_copy_service.cc:342] P 797fee1bbcb9413db379bc44ca2f43ec: Request end of tablet copy session 4ebd8eaf5f5c4d87918f3b07d6c25a20-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.119.252@KRBTEST.COM'} at 127.0.0.1:45256
I20260926 20:16:15.990476 5909 tablet_copy_service.cc:433] P 797fee1bbcb9413db379bc44ca2f43ec: ending tablet copy session 4ebd8eaf5f5c4d87918f3b07d6c25a20-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4ebd8eaf5f5c4d87918f3b07d6c25a20
I20260926 20:16:15.997814 6461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:16.001273 6655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:16.001478 6652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:16.001423 6653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:16.001879 6461 server_base.cc:1034] running on GCE node
I20260926 20:16:16.002074 6461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:16.002382 6461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:16.003533 6461 hybrid_clock.cc:648] HybridClock initialized: now 1790453776003520 us; error 33 us; skew 500 ppm
Sep 26 20:16:15 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453775, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:16.005468 6461 init.cc:377] Logged in from keytab as oryx/127.31.119.252@KRBTEST.COM (short username oryx)
I20260926 20:16:16.006309 6461 webserver.cc:460] Webserver started at http://127.31.119.252:41127/ using document root <none> and password file <none>
I20260926 20:16:16.006574 6461 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:16.006668 6461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:16.008147 6461 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:16:16.009256 6662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:16.009585 6461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:16:16.009698 6461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal
uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "1ad89dd1dbbd3b38dc375767493001e1"
server_key_iv: "91e1b55f3ae53b5bc2b9eb4cc42c6c48"
server_key_version: "encryptionkey@0"
I20260926 20:16:16.009847 6461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:16.021456 6216 heartbeater.cc:503] Master 127.31.119.254:46179 was elected leader, sending a full tablet report...
I20260926 20:16:16.031739 6461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:16.034539 6461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:16.039873 6461 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.252:43255
I20260926 20:16:16.039930 6714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.252:43255 every 8 connection(s)
I20260926 20:16:16.040727 6461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
I20260926 20:16:16.042831 6715 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:16:16.043972 6715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Bootstrap starting.
I20260926 20:16:16.056077 6715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:16:16.056442 6715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Bootstrap complete.
I20260926 20:16:16.057080 6715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } }
I20260926 20:16:16.057240 6715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4ebd8eaf5f5c4d87918f3b07d6c25a20, State: Initialized, Role: LEARNER
I20260926 20:16:16.057379 6715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } }
I20260926 20:16:16.057771 6718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:16.057885 6715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:16.058027 6718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:16:16.062624 6729 catalog_manager.cc:1259] Loaded cluster ID: 14dc690dc703444da8ff268d5ac1b4de
I20260926 20:16:16.062709 6729 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: loading cluster ID for follower catalog manager: success
I20260926 20:16:16.064896 6729 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: acquiring CA information for follower catalog manager: success
I20260926 20:16:16.066607 6729 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 20:16:16.069505 6641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:16.069869 6641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:16.069957 6641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:16.074438 6641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:16.074502 6641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:16.074596 6641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:16.080530 6641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab
--principal=oryx/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:35571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=34847
--tserver_master_addrs=127.31.119.254:46179,127.31.119.253:34189,127.31.119.252:43255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:16.081780 6641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:16.082894 6641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:16.093624 6735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:16.093636 6734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:16.093716 6641 server_base.cc:1034] running on GCE node
W20260926 20:16:16.093940 6737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:16.094365 6641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:16.095041 6641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:16.096246 6641 hybrid_clock.cc:648] HybridClock initialized: now 1790453776096207 us; error 51 us; skew 500 ppm
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:16.099434 6641 init.cc:377] Logged in from keytab as oryx/127.31.119.193@KRBTEST.COM (short username oryx)
I20260926 20:16:16.100426 6641 webserver.cc:460] Webserver started at http://127.31.119.193:34847/ using document root <none> and password file <none>
I20260926 20:16:16.100996 6641 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:16.101100 6641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:16.105034 6641 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:16.107295 6744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:16.108780 6641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:16.108888 6641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "7d4e11f6773e4724995df855fed7f7d4"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "7da887ad69c839183ee1107a87f5b3d0"
server_key_iv: "cd8bbcc1259ac4acc2c174278872155d"
server_key_version: "encryptionkey@0"
I20260926 20:16:16.109597 6641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:16.148522 6641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:16.152000 6641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:16.152231 6641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:16.152894 6641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:16.153919 6641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:16.153968 6641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:16.154035 6641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:16.154065 6641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:16.164110 6641 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:35571
I20260926 20:16:16.164148 6858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:35571 every 8 connection(s)
I20260926 20:16:16.165223 6641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:16.170301 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 6641
I20260926 20:16:16.170522 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 6086
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.253@KRBTEST.COM
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:16.179698 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:42921
--local_ip_for_outbound_sockets=127.31.119.194
--tserver_master_addrs=127.31.119.254:46179
--webserver_port=42859
--webserver_interface=127.31.119.194
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab
--principal=oryx/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.119.254:46179,127.31.119.253:34189
--tserver_master_addrs=127.31.119.254:46179,127.31.119.253:34189,127.31.119.252:43255 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:16:16.184295 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.167232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:50889 (local address 127.31.119.254:46179)
0926 20:16:16.167393 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.167398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.168435 (+ 1037us) server_negotiation.cc:407] Connection header received
0926 20:16:16.169708 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.169712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.169774 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.169876 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.172112 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.172956 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.174482 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.174771 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.174876 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.179083 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.179087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.179090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.179123 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.181467 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.182228 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.182232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.182234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.182327 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.182777 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.182780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.182782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.183025 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.183115 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.184004 (+ 889us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.184121 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:16:16.184295 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.168934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:48955 (local address 127.31.119.253:34189)
0926 20:16:16.169057 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.169062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.169125 (+ 63us) server_negotiation.cc:407] Connection header received
0926 20:16:16.169720 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.169724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.169778 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.169835 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.172181 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.172969 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.174677 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.175009 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.175110 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.179083 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.179087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.179090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.179123 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.181384 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.182228 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.182232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.182234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.182327 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.182777 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.182781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.182783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.183025 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.183131 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.183983 (+ 852us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.184121 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.252@KRBTEST.COM
I20260926 20:16:16.186255 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.168785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:34189 (local address 127.31.119.193:48955)
0926 20:16:16.169044 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.169058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.169496 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.170070 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.170081 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.171285 (+ 1204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:16.171995 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.172011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.173130 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.173137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.174543 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.174557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.175124 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.175128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.175320 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.176131 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:16.176164 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:16.178861 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:16.181531 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.181540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.181565 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.182101 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.182439 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.182444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.182446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.182664 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.183287 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:16.183290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:16.183583 (+ 293us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.184505 (+ 922us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.184973 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":84,"spinlock_wait_cycles":32512,"thread_start_us":83,"threads_started":1}
I20260926 20:16:16.186563 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.167716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.193:50889)
0926 20:16:16.168275 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.168329 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.169486 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.170070 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.170083 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.171301 (+ 1218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:16.171947 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.171969 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.173383 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.173386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.174303 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.174313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.175010 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.175017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.175332 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.176127 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:16.176155 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:16.178906 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:16.181574 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.181577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.181579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.182100 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.182459 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.182461 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.182463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.182664 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.183187 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:16.183195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:16.183812 (+ 617us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.184482 (+ 670us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.185020 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":273,"spinlock_wait_cycles":21632,"thread_start_us":112,"threads_started":1}
I20260926 20:16:16.188179 6860 heartbeater.cc:347] Connected to a master server at 127.31.119.254:46179
I20260926 20:16:16.188308 6861 heartbeater.cc:347] Connected to a master server at 127.31.119.253:34189
I20260926 20:16:16.188445 6860 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:16.188444 6861 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:16.189198 6860 heartbeater.cc:511] Master 127.31.119.254:46179 requested a full tablet report, sending...
I20260926 20:16:16.189198 6861 heartbeater.cc:511] Master 127.31.119.253:34189 requested a full tablet report, sending...
I20260926 20:16:16.191052 5889 ts_manager.cc:194] Re-registered known tserver with Master: 7d4e11f6773e4724995df855fed7f7d4 (127.31.119.193:35571)
I20260926 20:16:16.191273 6495 ts_manager.cc:194] Registered new tserver with Master: 7d4e11f6773e4724995df855fed7f7d4 (127.31.119.193:35571)
I20260926 20:16:16.191871 5889 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:50889
I20260926 20:16:16.192606 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.167386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:56191 (local address 127.31.119.252:43255)
0926 20:16:16.167641 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.167646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.168538 (+ 892us) server_negotiation.cc:407] Connection header received
0926 20:16:16.169720 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.169724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.169778 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.169838 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.178190 (+ 8352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.179358 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.181090 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.181546 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.181640 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.186394 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.186402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.186408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.186450 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.188633 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.189481 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.189485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.189489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.189595 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.190002 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.190005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.190007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.190273 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.190391 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.192168 (+ 1777us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.192343 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260926 20:16:16.193253 6864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.167715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43255 (local address 127.31.119.193:56191)
0926 20:16:16.168440 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.168455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.169495 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.177533 (+ 8038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.177536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.177754 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:16.178033 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.178042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.179618 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.179624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.180905 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.180921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.183110 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.183117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.183256 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.183775 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:16.183796 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:16.186168 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:16.188885 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.188890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.188894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.189213 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.189751 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.189757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.189762 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.189864 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.191772 (+ 1908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:16.191776 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:16.191909 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.192767 (+ 858us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.192971 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":48,"spinlock_wait_cycles":72064,"thread_start_us":158,"threads_started":1}
I20260926 20:16:16.194031 6859 heartbeater.cc:347] Connected to a master server at 127.31.119.252:43255
I20260926 20:16:16.194111 6859 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:16.194386 6859 heartbeater.cc:511] Master 127.31.119.252:43255 requested a full tablet report, sending...
I20260926 20:16:16.195529 6679 ts_manager.cc:194] Registered new tserver with Master: 7d4e11f6773e4724995df855fed7f7d4 (127.31.119.193:35571)
I20260926 20:16:16.252434 6351 heartbeater.cc:503] Master 127.31.119.254:46179 was elected leader, sending a full tablet report...
W20260926 20:16:16.304304 6867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:16.304627 6867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:16.304672 6867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:16.308987 6867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:16.309063 6867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:16.309127 6867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:16.315106 6867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab
--principal=oryx/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:42921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=42859
--tserver_master_addrs=127.31.119.254:46179,127.31.119.253:34189,127.31.119.252:43255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:16.316268 6867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:16.317373 6867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:16.327740 6872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:16.327740 6875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:16.327932 6873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:16.328133 6867 server_base.cc:1034] running on GCE node
I20260926 20:16:16.328594 6867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:16.329370 6867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:16.330556 6867 hybrid_clock.cc:648] HybridClock initialized: now 1790453776330525 us; error 55 us; skew 500 ppm
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:16.333899 6867 init.cc:377] Logged in from keytab as oryx/127.31.119.194@KRBTEST.COM (short username oryx)
I20260926 20:16:16.334950 6867 webserver.cc:460] Webserver started at http://127.31.119.194:42859/ using document root <none> and password file <none>
I20260926 20:16:16.335528 6867 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:16.335605 6867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:16.339946 6867 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:16:16.342288 6882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:16.343680 6867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:16.343923 6867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "9c0866b3d3c243a1b9349584dbbc8355"
format_stamp: "Formatted at 2026-09-26 20:16:14 on dist-test-slave-vnbh"
server_key: "af47d5da2dfb30db36a5d801026c7268"
server_key_iv: "1db5ab0fded5c0f077935f2329504f4d"
server_key_version: "encryptionkey@0"
I20260926 20:16:16.344511 6867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:16.354275 6867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:16.357800 6867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:16.357997 6867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:16.358611 6867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:16.359633 6867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:16.359684 6867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:16.359753 6867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:16.359783 6867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:16.369503 6867 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:42921
I20260926 20:16:16.369570 6996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:42921 every 8 connection(s)
I20260926 20:16:16.370595 6867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:16.377879 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 6867
I20260926 20:16:16.378082 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 6221
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.253@KRBTEST.COM
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.252@KRBTEST.COM
I20260926 20:16:16.386886 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:33481
--local_ip_for_outbound_sockets=127.31.119.195
--tserver_master_addrs=127.31.119.254:46179
--webserver_port=38291
--webserver_interface=127.31.119.195
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab
--principal=oryx/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.119.254:46179,127.31.119.253:34189
--tserver_master_addrs=127.31.119.254:46179,127.31.119.253:34189,127.31.119.252:43255 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:16:16.388175 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.374269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:56421 (local address 127.31.119.253:34189)
0926 20:16:16.374392 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.374396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.374510 (+ 114us) server_negotiation.cc:407] Connection header received
0926 20:16:16.375025 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.375029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.375090 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.375141 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.376507 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.377335 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.378667 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.379001 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.379096 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.383289 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.383293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.383296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.383330 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.385145 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.386008 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.386014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.386018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.386114 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.386549 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.386552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.386555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.386807 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.386876 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.387842 (+ 966us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.387995 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:16:16.388531 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.372565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:56377 (local address 127.31.119.254:46179)
0926 20:16:16.372709 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.372714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.374127 (+ 1413us) server_negotiation.cc:407] Connection header received
0926 20:16:16.375014 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.375018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.375082 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.375185 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.376484 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.377025 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.379153 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.379497 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.379588 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.383434 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.383441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.383447 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.383486 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.386553 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.387053 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.387060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.387064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.387157 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.387473 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.387480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.387485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.387739 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.387826 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.388196 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.388346 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:16:16.389183 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.372631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:56745 (local address 127.31.119.252:43255)
0926 20:16:16.372756 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.372761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.373813 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 20:16:16.375025 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.375029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.375085 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.375141 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.376508 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.377330 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.380381 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.380855 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.380954 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.383859 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.383865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.383872 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.383906 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.386715 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.387549 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.387553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.387555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.387647 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.388022 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.388026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.388027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.388238 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.388315 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.388901 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.389039 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:16:16.390102 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.372984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.194:56377)
0926 20:16:16.374037 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.374052 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.374784 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.375369 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.375377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.375804 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:16.376333 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.376347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.377168 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.377171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.379029 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.379039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.379775 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.379780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.379885 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.380409 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:16.380436 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:16.383273 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:16.386688 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.386691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.386693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.386927 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.387271 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.387277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.387282 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.387374 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.387890 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:16.387895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:16.388004 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.388759 (+ 755us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.388947 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":119,"spinlock_wait_cycles":33024,"thread_start_us":145,"threads_started":1}
I20260926 20:16:16.390113 7003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.374137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:34189 (local address 127.31.119.194:56421)
0926 20:16:16.374396 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.374420 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.374826 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.375369 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.375378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.375936 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:16.376359 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.376368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.377475 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.377478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.378535 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.378546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.379300 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.379304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.379486 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.380409 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:16.380441 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:16.383094 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:16.385385 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.385406 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.385431 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.385863 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.386238 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.386241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.386243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.386433 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.387075 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:16.387085 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:16.387548 (+ 463us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.388449 (+ 901us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.388880 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":118,"spinlock_wait_cycles":81280,"thread_start_us":87,"threads_started":1}
I20260926 20:16:16.390113 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.372984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43255 (local address 127.31.119.194:56745)
0926 20:16:16.373599 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.373674 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.374826 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.375369 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.375378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.375936 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:16.376358 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.376368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.377475 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.377478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.380229 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.380243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.381126 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.381132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.381265 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.381803 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:16.381823 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:16.383667 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:16.386840 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.386845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.386850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.387416 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.387767 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.387773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.387778 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.387861 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.388632 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:16.388634 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:16.388718 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.389357 (+ 639us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.389495 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":204,"spinlock_wait_cycles":6272,"thread_start_us":793,"threads_started":1}
I20260926 20:16:16.391729 6999 heartbeater.cc:347] Connected to a master server at 127.31.119.253:34189
I20260926 20:16:16.391788 6998 heartbeater.cc:347] Connected to a master server at 127.31.119.254:46179
I20260926 20:16:16.391824 6997 heartbeater.cc:347] Connected to a master server at 127.31.119.252:43255
I20260926 20:16:16.392021 6999 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:16.392011 6998 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:16.392011 6997 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:16.392684 6997 heartbeater.cc:511] Master 127.31.119.252:43255 requested a full tablet report, sending...
I20260926 20:16:16.392681 6998 heartbeater.cc:511] Master 127.31.119.254:46179 requested a full tablet report, sending...
I20260926 20:16:16.392681 6999 heartbeater.cc:511] Master 127.31.119.253:34189 requested a full tablet report, sending...
I20260926 20:16:16.393975 5889 ts_manager.cc:194] Re-registered known tserver with Master: 9c0866b3d3c243a1b9349584dbbc8355 (127.31.119.194:42921)
I20260926 20:16:16.394047 6679 ts_manager.cc:194] Registered new tserver with Master: 9c0866b3d3c243a1b9349584dbbc8355 (127.31.119.194:42921)
I20260926 20:16:16.394179 6495 ts_manager.cc:194] Registered new tserver with Master: 9c0866b3d3c243a1b9349584dbbc8355 (127.31.119.194:42921)
I20260926 20:16:16.394805 5889 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:56377
W20260926 20:16:16.509423 7004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:16.509770 7004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:16.509863 7004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:16.514187 7004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:16.514264 7004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:16.514359 7004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:16.519994 7004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab
--principal=oryx/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:33481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=38291
--tserver_master_addrs=127.31.119.254:46179,127.31.119.253:34189,127.31.119.252:43255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:16.521214 7004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:16.522415 7004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:16.533313 7004 server_base.cc:1034] running on GCE node
W20260926 20:16:16.533229 7012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:16.533231 7009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:16.533231 7010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:16.533955 7004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:16.534636 7004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:16.535837 7004 hybrid_clock.cc:648] HybridClock initialized: now 1790453776535824 us; error 46 us; skew 500 ppm
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:16.538949 7004 init.cc:377] Logged in from keytab as oryx/127.31.119.195@KRBTEST.COM (short username oryx)
I20260926 20:16:16.539937 7004 webserver.cc:460] Webserver started at http://127.31.119.195:38291/ using document root <none> and password file <none>
I20260926 20:16:16.540544 7004 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:16.540665 7004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:16.545871 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.541522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43255 (local address 127.0.0.1:57004)
0926 20:16:16.541896 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.541907 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.542014 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.542334 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.542337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.542358 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:16.542456 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.542461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.543530 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.543534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.544437 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.544452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.545435 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.545438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.545561 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.545586 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.545642 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.545710 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":77,"threads_started":1}
I20260926 20:16:16.545878 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.541658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.31.119.252:43255)
0926 20:16:16.541786 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.541791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.541988 (+ 197us) server_negotiation.cc:407] Connection header received
0926 20:16:16.542102 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.542106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.542173 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.542290 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:16.542589 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.543367 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.544607 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.545239 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.545466 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.545617 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.545674 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.545747 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:16:16.545832 7004 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:16:16.546625 6687 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 20:16:16.547067 6718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 797fee1bbcb9413db379bc44ca2f43ec. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:16.547230 6718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:16:16.548878 7020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:16.550464 7004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:16.550629 7004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "4dea1409891644e685da300de2183ef3"
format_stamp: "Formatted at 2026-09-26 20:16:15 on dist-test-slave-vnbh"
server_key: "9742d84ab463669df28a1882142f8d41"
server_key_iv: "79a2b95397412c4ab43aa3101a746b4a"
server_key_version: "encryptionkey@0"
I20260926 20:16:16.551229 7004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:16.579466 7004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:16.582969 7004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:16.583151 7004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:16.583726 7004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:16.584657 7004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:16.584730 7004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:16.584805 7004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:16.584856 7004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:16.595112 7004 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:33481
I20260926 20:16:16.595144 7133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:33481 every 8 connection(s)
I20260926 20:16:16.596233 7004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:16.597461 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 7004
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.252@KRBTEST.COM
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.253@KRBTEST.COM
I20260926 20:16:16.612154 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.600214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.31.119.254:46179)
0926 20:16:16.600552 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.600557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.600575 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:16:16.600624 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.600628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.600675 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.600773 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.601773 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.602574 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.603960 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.604281 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.604356 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.606671 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.606674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.606677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.606725 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.609448 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.609964 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.609967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.609969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.610080 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.610776 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.610782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.610785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.610992 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.611126 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.611810 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.611943 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260926 20:16:16.613478 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.599573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.31.119.252:43255)
0926 20:16:16.600492 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.600499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.600514 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:16.600662 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.600665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.600716 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.600791 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.601976 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.602748 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.603942 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.604311 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.604400 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.606601 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.606605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.606608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.606636 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.608596 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.609145 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.609149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.609151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.609240 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.609606 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.609611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.609615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.609851 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.609976 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.613094 (+ 3118us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.613303 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":834,"thread_start_us":65,"threads_started":1}
I20260926 20:16:16.615588 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.598442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:58697 (local address 127.31.119.252:43255)
0926 20:16:16.598750 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.598755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.600981 (+ 2226us) server_negotiation.cc:407] Connection header received
0926 20:16:16.604095 (+ 3114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.604098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.604147 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.604195 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.604792 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.605560 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.607433 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.607775 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.607849 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.611196 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.611200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.611203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.611226 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.612917 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.613624 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.613628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.613629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.613714 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.614309 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.614313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.614314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.614541 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.614646 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.615360 (+ 714us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.615473 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
Sep 26 20:16:16 dist-test-slave-vnbh krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453776, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:16.618497 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.598905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43255 (local address 127.31.119.195:58697)
0926 20:16:16.600795 (+ 1890us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.600870 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.602067 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.604325 (+ 2258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.604329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.604546 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:16.604654 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.604664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.605741 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.605747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.607293 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.607305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.608084 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.608092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.608381 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.608909 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:16.608933 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:16.610996 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:16.613039 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.613045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.613064 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.613509 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.613883 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.613886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.613887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.614129 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.614709 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:16.614716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:16.615060 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.616299 (+ 1239us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.616739 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"spinlock_wait_cycles":3328,"thread_start_us":707,"threads_started":1}
I20260926 20:16:16.616501 5889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:16:16.620031 7134 heartbeater.cc:347] Connected to a master server at 127.31.119.252:43255
I20260926 20:16:16.620278 7134 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:16.620548 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.598603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:46717 (local address 127.31.119.253:34189)
0926 20:16:16.598844 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.598848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.601596 (+ 2748us) server_negotiation.cc:407] Connection header received
0926 20:16:16.602334 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.602336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.602367 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.602420 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.604055 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.604719 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.608561 (+ 3842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.608959 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.609042 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.615833 (+ 6791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.615838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.615844 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.615879 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.617585 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.619291 (+ 1706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.619295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.619296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.619373 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.619690 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.619694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.619695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.619877 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.619959 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.620316 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.620408 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"spinlock_wait_cycles":1280}
I20260926 20:16:16.620474 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.599950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.31.119.253:34189)
0926 20:16:16.601342 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.601346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.602003 (+ 657us) server_negotiation.cc:407] Connection header received
0926 20:16:16.602049 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.602052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.602099 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.602177 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.603264 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.604681 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.606033 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.606450 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.606579 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.612822 (+ 6243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.612826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.612829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.614314 (+ 1485us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.618132 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.618671 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.618674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.618676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.618767 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.619117 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.619120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.619123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.619365 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.619489 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.620159 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.620299 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1294,"thread_start_us":83,"threads_started":1}
I20260926 20:16:16.620857 7134 heartbeater.cc:511] Master 127.31.119.252:43255 requested a full tablet report, sending...
I20260926 20:16:16.620882 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.598905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:34189 (local address 127.31.119.195:46717)
0926 20:16:16.601493 (+ 2588us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.601508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.602266 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.602640 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.602649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.603228 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:16.603916 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.603939 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.606880 (+ 2941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.606886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.608017 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.608034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.609842 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.609846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.609960 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.610611 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:16.610626 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:16.615653 (+ 5027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:16.618916 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.618921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.618926 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.619184 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.619475 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.619478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.619480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.619583 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.620023 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:16.620025 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:16.620128 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.620516 (+ 388us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.620673 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2491,"mutex_wait_us":26,"spinlock_wait_cycles":23296,"thread_start_us":186,"threads_started":1}
W20260926 20:16:16.621150 5889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:16:16.622166 7136 heartbeater.cc:347] Connected to a master server at 127.31.119.253:34189
I20260926 20:16:16.622249 7136 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:16.622430 7136 heartbeater.cc:511] Master 127.31.119.253:34189 requested a full tablet report, sending...
I20260926 20:16:16.622478 6679 ts_manager.cc:194] Registered new tserver with Master: 4dea1409891644e685da300de2183ef3 (127.31.119.195:33481)
I20260926 20:16:16.623054 6494 ts_manager.cc:194] Registered new tserver with Master: 4dea1409891644e685da300de2183ef3 (127.31.119.195:33481)
I20260926 20:16:16.624361 6454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: NON_VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: true } } }
I20260926 20:16:16.624504 6454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:16.625865 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.600321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:45953 (local address 127.31.119.254:46179)
0926 20:16:16.600552 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.600565 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.605259 (+ 4694us) server_negotiation.cc:407] Connection header received
0926 20:16:16.607622 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.607625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.607680 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.607765 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.610551 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.611324 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.613157 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.613524 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.613636 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.617101 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.617108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.617111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.617139 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.621917 (+ 4778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.623650 (+ 1733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.623654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.623656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.623753 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.624729 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.624733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.624735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.624979 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.625092 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.625562 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.625708 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":112,"threads_started":1}
I20260926 20:16:16.626117 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.600177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.195:45953)
0926 20:16:16.605162 (+ 4985us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.605176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.605261 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.610045 (+ 4784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.610051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.610260 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:16.610389 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.610397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.611869 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.611873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.613013 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.613025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.613773 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.613777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.613867 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.614735 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:16.614754 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:16.616963 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:16.623277 (+ 6314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.623281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.623282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.623513 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.623862 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:16.623864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:16.623866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:16.623953 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:16.625107 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:16.625111 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:16.625210 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.625475 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.625640 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4901,"spinlock_wait_cycles":896,"thread_start_us":75,"threads_started":1}
I20260926 20:16:16.627206 7135 heartbeater.cc:347] Connected to a master server at 127.31.119.254:46179
I20260926 20:16:16.627445 7135 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:16.627640 7135 heartbeater.cc:511] Master 127.31.119.254:46179 requested a full tablet report, sending...
I20260926 20:16:16.628331 5888 ts_manager.cc:194] Re-registered known tserver with Master: 4dea1409891644e685da300de2183ef3 (127.31.119.195:33481)
I20260926 20:16:16.628753 7152 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec: attempting to promote NON_VOTER 4ebd8eaf5f5c4d87918f3b07d6c25a20 to VOTER
I20260926 20:16:16.629105 7152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } }
I20260926 20:16:16.629385 5888 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:45953
I20260926 20:16:16.632071 6688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 LEARNER]: Refusing update from remote peer 797fee1bbcb9413db379bc44ca2f43ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 20:16:16.632349 6504 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 FOLLOWER]: Refusing update from remote peer 797fee1bbcb9413db379bc44ca2f43ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 20:16:16.632606 7153 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 20:16:16.633090 7153 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 20:16:16.634961 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4ebd8eaf5f5c4d87918f3b07d6c25a20 (127.31.119.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:16.635450 6504 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4ebd8eaf5f5c4d87918f3b07d6c25a20 (127.31.119.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:16.635397 6687 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4ebd8eaf5f5c4d87918f3b07d6c25a20 (127.31.119.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:16.636624 7152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:16.636775 7152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:16.636941 7152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:16.637043 7152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:16.637005 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:16.637247 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:16.637266 6718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:16.637395 6718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:16:16.641428 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "797fee1bbcb9413db379bc44ca2f43ec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:16.641584 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:16:16.644256 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.639114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:33481 (local address 127.0.0.1:59986)
0926 20:16:16.639244 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.639259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.639337 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.640255 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.640258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.640274 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:16.640392 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.640403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.641799 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.641802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.642968 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.642976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.643853 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.643856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.643949 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.643970 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.644028 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.644117 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":31}
I20260926 20:16:16.644308 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.639294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.31.119.195:33481)
0926 20:16:16.639696 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.639709 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.639734 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:16:16.639815 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.639819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.640080 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.640292 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:16.640521 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.641664 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.643114 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.643720 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.643858 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.644007 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.644081 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.644175 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":117,"threads_started":1}
I20260926 20:16:16.645745 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.639779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35571 (local address 127.0.0.1:55312)
0926 20:16:16.640006 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.640021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.640111 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.641003 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.641006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.641025 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:16.641129 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.641135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.642892 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.642896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.644213 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.644223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.645402 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.645407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.645509 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.645532 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.645586 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.645638 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":71,"thread_start_us":102,"threads_started":1}
I20260926 20:16:16.646374 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.640496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:42921 (local address 127.0.0.1:57628)
0926 20:16:16.640747 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.640760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.640846 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.641924 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.641927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.641946 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:16.642042 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.642047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.643198 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.643201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.644345 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.644353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.645315 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.645319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.645974 (+ 655us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.646123 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.646193 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.646268 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":3,"thread_start_us":126,"threads_started":1}
I20260926 20:16:16.646538 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.639941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.31.119.193:35571)
0926 20:16:16.640556 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.640572 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.640594 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:16:16.640668 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.640673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.640882 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.641034 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:16.641223 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.642262 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.644360 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.645169 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.646108 (+ 939us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.646237 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.646288 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.646382 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":80,"threads_started":1}
I20260926 20:16:16.647440 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.640945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.31.119.194:42921)
0926 20:16:16.641478 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.641489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.641521 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:16:16.641594 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.641601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.641795 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.641946 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:16.642170 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.643088 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.644539 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.645152 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.645295 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.645472 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.646565 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.647268 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":113,"threads_started":1}
I20260926 20:16:16.647763 7068 tablet_service.cc:1467] Processing CreateTablet for tablet 6175e6b210714e83aff98ea3b3266112 (DEFAULT_TABLE table=test-table [id=9c4d66b3efcf438cb1d3a5c1eed82d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:16.648941 7068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6175e6b210714e83aff98ea3b3266112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:16.649415 6793 tablet_service.cc:1467] Processing CreateTablet for tablet 6175e6b210714e83aff98ea3b3266112 (DEFAULT_TABLE table=test-table [id=9c4d66b3efcf438cb1d3a5c1eed82d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:16.650408 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6175e6b210714e83aff98ea3b3266112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:16.651692 6931 tablet_service.cc:1467] Processing CreateTablet for tablet 6175e6b210714e83aff98ea3b3266112 (DEFAULT_TABLE table=test-table [id=9c4d66b3efcf438cb1d3a5c1eed82d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:16.652403 6931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6175e6b210714e83aff98ea3b3266112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:16.657575 7168 tablet_bootstrap.cc:492] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: Bootstrap starting.
I20260926 20:16:16.658813 7169 tablet_bootstrap.cc:492] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: Bootstrap starting.
I20260926 20:16:16.659559 7170 tablet_bootstrap.cc:492] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: Bootstrap starting.
I20260926 20:16:16.660574 7168 tablet_bootstrap.cc:654] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:16.661190 7169 tablet_bootstrap.cc:654] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:16.661690 7170 tablet_bootstrap.cc:654] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:16.661923 7168 log.cc:824] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:16.662050 7169 log.cc:824] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:16.662456 7170 log.cc:824] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:16.663980 7168 tablet_bootstrap.cc:492] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: No bootstrap required, opened a new log
I20260926 20:16:16.664242 7168 ts_tablet_manager.cc:1397] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 20:16:16.664500 7170 tablet_bootstrap.cc:492] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: No bootstrap required, opened a new log
I20260926 20:16:16.664698 7170 ts_tablet_manager.cc:1397] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:16:16.666791 7169 tablet_bootstrap.cc:492] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: No bootstrap required, opened a new log
I20260926 20:16:16.666994 7169 ts_tablet_manager.cc:1397] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 20:16:16.668751 7170 raft_consensus.cc:348] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } }
I20260926 20:16:16.669036 7170 raft_consensus.cc:374] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:16.668874 7168 raft_consensus.cc:348] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } }
I20260926 20:16:16.669096 7170 raft_consensus.cc:729] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0866b3d3c243a1b9349584dbbc8355, State: Initialized, Role: FOLLOWER
I20260926 20:16:16.669139 7168 raft_consensus.cc:374] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:16.669202 7168 raft_consensus.cc:729] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dea1409891644e685da300de2183ef3, State: Initialized, Role: FOLLOWER
I20260926 20:16:16.669618 7170 consensus_queue.cc:260] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } }
I20260926 20:16:16.669669 7168 consensus_queue.cc:260] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } }
I20260926 20:16:16.670763 7168 ts_tablet_manager.cc:1428] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260926 20:16:16.670785 7170 ts_tablet_manager.cc:1428] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:16:16.671088 7135 heartbeater.cc:503] Master 127.31.119.254:46179 was elected leader, sending a full tablet report...
I20260926 20:16:16.671849 7169 raft_consensus.cc:348] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } }
I20260926 20:16:16.672129 7169 raft_consensus.cc:374] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:16.672188 7169 raft_consensus.cc:729] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4e11f6773e4724995df855fed7f7d4, State: Initialized, Role: FOLLOWER
I20260926 20:16:16.672667 7169 consensus_queue.cc:260] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } }
I20260926 20:16:16.673768 7169 ts_tablet_manager.cc:1428] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 20:16:16.674275 7175 raft_consensus.cc:482] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:16:16.674933 6998 heartbeater.cc:503] Master 127.31.119.254:46179 was elected leader, sending a full tablet report...
I20260926 20:16:16.675791 6860 heartbeater.cc:503] Master 127.31.119.254:46179 was elected leader, sending a full tablet report...
I20260926 20:16:16.676584 7175 raft_consensus.cc:504] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } }
I20260926 20:16:16.678530 7175 leader_election.cc:288] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4e11f6773e4724995df855fed7f7d4 (127.31.119.193:35571), 9c0866b3d3c243a1b9349584dbbc8355 (127.31.119.194:42921)
I20260926 20:16:16.683737 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.678772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:42921 (local address 127.31.119.195:49355)
0926 20:16:16.678902 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.678919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.679000 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.679511 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.679514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.679546 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:16.679728 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.679735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.680749 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.680752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.682139 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.682150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.683150 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.683154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.683382 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.683423 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.683497 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.683598 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":135}
I20260926 20:16:16.684442 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.679097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:49355 (local address 127.31.119.194:42921)
0926 20:16:16.679210 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.679215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.679227 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:16:16.679271 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.679274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.679321 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.679389 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:16.679857 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.680627 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.682292 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.682916 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.683032 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.683088 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.684242 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.684327 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:16:16.685015 6951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6175e6b210714e83aff98ea3b3266112" candidate_uuid: "4dea1409891644e685da300de2183ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0866b3d3c243a1b9349584dbbc8355" is_pre_election: true
I20260926 20:16:16.685375 6951 raft_consensus.cc:2454] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dea1409891644e685da300de2183ef3 in term 0.
I20260926 20:16:16.685879 7021 leader_election.cc:302] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dea1409891644e685da300de2183ef3, 9c0866b3d3c243a1b9349584dbbc8355; no voters:
I20260926 20:16:16.687297 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.679189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:46687 (local address 127.31.119.193:35571)
0926 20:16:16.679296 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.679300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.679312 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:16:16.679353 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.679359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.679403 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.679466 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:16.680086 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.680785 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.682317 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.682916 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.683044 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.683088 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.687064 (+ 3976us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.687180 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:16:16.687562 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.679048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35571 (local address 127.31.119.195:46687)
0926 20:16:16.679141 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:16.679155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:16.679230 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:16.679554 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:16.679557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:16.679569 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:16.679730 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.679735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.680895 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.680899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.682192 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:16.682202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.683150 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.683154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:16.683377 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.683423 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:16:16.683497 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:16:16.687457 (+ 3960us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":4025}
I20260926 20:16:16.687736 7176 raft_consensus.cc:2781] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:16:16.688115 6813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6175e6b210714e83aff98ea3b3266112" candidate_uuid: "4dea1409891644e685da300de2183ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4e11f6773e4724995df855fed7f7d4" is_pre_election: true
I20260926 20:16:16.688395 6813 raft_consensus.cc:2454] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dea1409891644e685da300de2183ef3 in term 0.
I20260926 20:16:16.687853 7176 raft_consensus.cc:482] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:16:16.688664 7176 raft_consensus.cc:3037] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:16.689908 7176 raft_consensus.cc:504] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } }
I20260926 20:16:16.690339 7176 leader_election.cc:288] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [CANDIDATE]: Term 1 election: Requested vote from peers 7d4e11f6773e4724995df855fed7f7d4 (127.31.119.193:35571), 9c0866b3d3c243a1b9349584dbbc8355 (127.31.119.194:42921)
I20260926 20:16:16.690724 6813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6175e6b210714e83aff98ea3b3266112" candidate_uuid: "4dea1409891644e685da300de2183ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4e11f6773e4724995df855fed7f7d4"
I20260926 20:16:16.690872 6813 raft_consensus.cc:3037] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:16.691926 6813 raft_consensus.cc:2454] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dea1409891644e685da300de2183ef3 in term 1.
I20260926 20:16:16.692188 6951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6175e6b210714e83aff98ea3b3266112" candidate_uuid: "4dea1409891644e685da300de2183ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0866b3d3c243a1b9349584dbbc8355"
I20260926 20:16:16.692260 7023 leader_election.cc:302] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dea1409891644e685da300de2183ef3, 7d4e11f6773e4724995df855fed7f7d4; no voters:
I20260926 20:16:16.692327 6951 raft_consensus.cc:3037] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:16.692454 7176 raft_consensus.cc:2781] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:16.692567 7176 raft_consensus.cc:686] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 1 LEADER]: Becoming Leader. State: Replica: 4dea1409891644e685da300de2183ef3, State: Running, Role: LEADER
I20260926 20:16:16.692925 7176 consensus_queue.cc:237] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } }
I20260926 20:16:16.693684 6951 raft_consensus.cc:2454] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dea1409891644e685da300de2183ef3 in term 1.
I20260926 20:16:16.697019 5888 catalog_manager.cc:5495] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dea1409891644e685da300de2183ef3 (127.31.119.195). New cstate: current_term: 1 leader_uuid: "4dea1409891644e685da300de2183ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } health_report { overall_health: UNKNOWN } } }
I20260926 20:16:16.711021 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.707360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.31.119.195:33481)
0926 20:16:16.707490 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.707494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.707509 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:16.707639 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.707642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.707698 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.707790 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:16.708025 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.708509 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.709415 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.709665 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.709756 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.710079 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:16.710557 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:16.710692 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.710759 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.710824 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:16:16.717635 6951 raft_consensus.cc:1261] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355 [term 1 FOLLOWER]: Refusing update from remote peer 4dea1409891644e685da300de2183ef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:16.717635 6813 raft_consensus.cc:1261] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4 [term 1 FOLLOWER]: Refusing update from remote peer 4dea1409891644e685da300de2183ef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:16.718372 7175 consensus_queue.cc:1035] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:16.718577 7176 consensus_queue.cc:1035] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:16.728355 7181 mvcc.cc:205] Tried to move back new op lower bound from 7333698669427232768 to 7333698669338021888. Current Snapshot: MvccSnapshot[applied={T|T < 7333698669427232768}]
I20260926 20:16:16.741528 5888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:16.741747 5888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:16.746572 5888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 797fee1bbcb9413db379bc44ca2f43ec: Sending DeleteTablet for 3 replicas of tablet 6175e6b210714e83aff98ea3b3266112
I20260926 20:16:16.747608 7068 tablet_service.cc:1514] Processing DeleteTablet for tablet 6175e6b210714e83aff98ea3b3266112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:16 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:59986
I20260926 20:16:16.747757 6931 tablet_service.cc:1514] Processing DeleteTablet for tablet 6175e6b210714e83aff98ea3b3266112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:16 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:57628
I20260926 20:16:16.748183 6793 tablet_service.cc:1514] Processing DeleteTablet for tablet 6175e6b210714e83aff98ea3b3266112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:16 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:55312
I20260926 20:16:16.748739 7190 tablet_replica.cc:330] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: stopping tablet replica
I20260926 20:16:16.748739 7192 tablet_replica.cc:330] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: stopping tablet replica
I20260926 20:16:16.749128 7190 raft_consensus.cc:2229] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:16:16.749128 7192 raft_consensus.cc:2229] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:16.749651 7192 raft_consensus.cc:2258] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:16.750813 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 5856
I20260926 20:16:16.750815 7190 raft_consensus.cc:2258] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:16.751905 7191 tablet_replica.cc:330] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: stopping tablet replica
I20260926 20:16:16.752301 7191 raft_consensus.cc:2229] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:16.752621 7191 raft_consensus.cc:2258] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:16.753520 7190 ts_tablet_manager.cc:1905] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:16.754441 7191 ts_tablet_manager.cc:1905] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:16.757162 7191 ts_tablet_manager.cc:1918] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:16.757256 7191 log.cc:1196] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/6175e6b210714e83aff98ea3b3266112
I20260926 20:16:16.757613 7191 ts_tablet_manager.cc:1939] T 6175e6b210714e83aff98ea3b3266112 P 9c0866b3d3c243a1b9349584dbbc8355: Deleting consensus metadata
I20260926 20:16:16.757829 7190 ts_tablet_manager.cc:1918] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:16.757944 7190 log.cc:1196] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/6175e6b210714e83aff98ea3b3266112
I20260926 20:16:16.758347 7190 ts_tablet_manager.cc:1939] T 6175e6b210714e83aff98ea3b3266112 P 4dea1409891644e685da300de2183ef3: Deleting consensus metadata
W20260926 20:16:16.759625 6883 connection.cc:535] server connection from 127.0.0.1:57628 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 20:16:16.760128 7024 connection.cc:535] server connection from 127.0.0.1:59986 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:16:16.761554 7192 ts_tablet_manager.cc:1905] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:16.764245 7192 ts_tablet_manager.cc:1918] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:16.764357 7192 log.cc:1196] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/6175e6b210714e83aff98ea3b3266112
I20260926 20:16:16.764675 7192 ts_tablet_manager.cc:1939] T 6175e6b210714e83aff98ea3b3266112 P 7d4e11f6773e4724995df855fed7f7d4: Deleting consensus metadata
W20260926 20:16:16.765878 6746 connection.cc:441] server connection from 127.0.0.1:55312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55312 (request call id 1) could send its response
I20260926 20:16:16.770319 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.750354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.31.119.253:34189)
0926 20:16:16.750464 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.750469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.753092 (+ 2623us) server_negotiation.cc:407] Connection header received
0926 20:16:16.759139 (+ 6047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.759143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.759200 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.759257 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.760452 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.761371 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.763640 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.764020 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.764123 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.766402 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.766406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.766409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.766440 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.768305 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.768767 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.768771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.768773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.768865 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.769319 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.769323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.769325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.769545 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.769658 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.770008 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.770151 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:16:16.776075 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.765775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.31.119.252:43255)
0926 20:16:16.765929 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.765936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.766576 (+ 640us) server_negotiation.cc:407] Connection header received
0926 20:16:16.766822 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.766825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.766877 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.766928 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.768003 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.768800 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.770087 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.770437 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.770516 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.772247 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.772251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.772254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.772284 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.774246 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.774710 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.774714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.774715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.774800 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.775106 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.775109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.775111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.775306 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.775407 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.775773 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.775899 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:16:16.776176 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.764925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.31.119.253:34189)
0926 20:16:16.765063 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.765069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.766611 (+ 1542us) server_negotiation.cc:407] Connection header received
0926 20:16:16.766800 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.766804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.766866 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.766921 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.767994 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.768831 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.770147 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.770462 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.770597 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.772509 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.772514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.772519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.772546 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.774249 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.774883 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.774887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.774889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.774966 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.775262 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.775265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.775267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.775460 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.775568 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.775898 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.776015 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:16:16.781625 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:16.750170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.31.119.252:43255)
0926 20:16:16.750315 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:16.750321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:16.772956 (+ 22635us) server_negotiation.cc:407] Connection header received
0926 20:16:16.773174 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:16.773177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:16.773235 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:16.773334 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:16.774161 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.774937 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.776059 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:16.776385 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:16.776478 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:16.778296 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:16.778300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:16.778303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:16.778328 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:16.779794 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.780228 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.780231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.780233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.780315 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:16.780603 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:16.780606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:16.780608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:16.780816 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:16.780949 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:16.781256 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 20:16:16.781438 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:16:17.733440 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:17.732838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.194:37581)
0926 20:16:17.733125 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:17.733269 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260926 20:16:17.733477 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:17.733006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.195:56175)
0926 20:16:17.733233 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:17.733366 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
W20260926 20:16:17.733768 6998 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:46179 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
W20260926 20:16:17.733942 7135 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:46179 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
I20260926 20:16:17.735239 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:17.734799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.193:51887)
0926 20:16:17.735035 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:17.735126 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
W20260926 20:16:17.735555 6860 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:46179 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
I20260926 20:16:18.247677 7204 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 797fee1bbcb9413db379bc44ca2f43ec)
I20260926 20:16:18.247664 7205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 797fee1bbcb9413db379bc44ca2f43ec)
I20260926 20:16:18.247916 7205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } }
I20260926 20:16:18.247915 7204 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } }
I20260926 20:16:18.249200 7204 leader_election.cc:288] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 797fee1bbcb9413db379bc44ca2f43ec (127.31.119.254:46179), 4ebd8eaf5f5c4d87918f3b07d6c25a20 (127.31.119.252:43255)
I20260926 20:16:18.249188 7205 leader_election.cc:288] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 797fee1bbcb9413db379bc44ca2f43ec (127.31.119.254:46179), c5f033f13aa84455b5f010f47419e62a (127.31.119.253:34189)
I20260926 20:16:18.250000 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.249467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:60730)
0926 20:16:18.249764 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.249856 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
W20260926 20:16:18.250259 6665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
I20260926 20:16:18.250759 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.249386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:60728)
0926 20:16:18.250011 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.250091 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":66,"threads_started":1}
W20260926 20:16:18.250929 6481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
I20260926 20:16:18.251312 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.251067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:60742)
0926 20:16:18.251180 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.251237 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 20:16:18.251607 6665 leader_election.cc:334] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 797fee1bbcb9413db379bc44ca2f43ec (127.31.119.254:46179): Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
I20260926 20:16:18.251995 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.251712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:60758)
0926 20:16:18.251879 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.251930 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260926 20:16:18.252097 6481 leader_election.cc:334] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 797fee1bbcb9413db379bc44ca2f43ec (127.31.119.254:46179): Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
I20260926 20:16:18.254393 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.249798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:34189 (local address 127.0.0.1:36370)
0926 20:16:18.250158 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.250171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:18.250279 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:18.251324 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:18.251330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:18.251354 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:18.251467 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.251474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.252409 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.252414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.253227 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.253236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.253950 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.253953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.254047 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.254077 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:16:18.254180 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:16:18.254259 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":48,"threads_started":1}
I20260926 20:16:18.254408 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.249949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36370 (local address 127.31.119.253:34189)
0926 20:16:18.250912 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:18.250916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:18.250932 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:16:18.251028 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:18.251032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:18.251090 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:18.251184 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:18.251601 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.252298 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.253397 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.253828 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.253967 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.254021 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:18.254197 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 20:16:18.254265 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":68,"threads_started":1}
I20260926 20:16:18.254904 6504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c5f033f13aa84455b5f010f47419e62a" is_pre_election: true
I20260926 20:16:18.255194 6504 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ebd8eaf5f5c4d87918f3b07d6c25a20 in term 1.
I20260926 20:16:18.255673 6665 leader_election.cc:302] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ebd8eaf5f5c4d87918f3b07d6c25a20, c5f033f13aa84455b5f010f47419e62a; no voters: 797fee1bbcb9413db379bc44ca2f43ec
I20260926 20:16:18.255934 7205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:16:18.256114 7205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 797fee1bbcb9413db379bc44ca2f43ec)
I20260926 20:16:18.256168 7205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:16:18.256166 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.249761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43255 (local address 127.0.0.1:57046)
0926 20:16:18.251198 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.251214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:18.251309 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:18.251620 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:18.251626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:18.251646 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:18.251741 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.251747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.252961 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.252967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.253881 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.253891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.255587 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.255591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.255702 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.255729 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:16:18.255798 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:16:18.255951 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"thread_start_us":71,"threads_started":1}
I20260926 20:16:18.256237 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.249949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.31.119.252:43255)
0926 20:16:18.250201 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:18.250205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:18.251379 (+ 1174us) server_negotiation.cc:407] Connection header received
0926 20:16:18.251431 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:18.251433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:18.251495 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:18.251844 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:18.251936 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.252585 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.254006 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.255457 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.255697 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.255767 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:18.255823 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 20:16:18.255906 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":49,"threads_started":1}
I20260926 20:16:18.256739 6687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5f033f13aa84455b5f010f47419e62a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" is_pre_election: true
I20260926 20:16:18.257395 7205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } }
I20260926 20:16:18.257647 6687 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5f033f13aa84455b5f010f47419e62a in current term 2: Already voted for candidate 4ebd8eaf5f5c4d87918f3b07d6c25a20 in this term.
I20260926 20:16:18.257819 7205 leader_election.cc:288] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [CANDIDATE]: Term 2 election: Requested vote from peers 797fee1bbcb9413db379bc44ca2f43ec (127.31.119.254:46179), c5f033f13aa84455b5f010f47419e62a (127.31.119.253:34189)
I20260926 20:16:18.258240 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.258017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:60768)
0926 20:16:18.258139 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.258186 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 20:16:18.258371 6480 leader_election.cc:302] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5f033f13aa84455b5f010f47419e62a; no voters: 4ebd8eaf5f5c4d87918f3b07d6c25a20, 797fee1bbcb9413db379bc44ca2f43ec
I20260926 20:16:18.258628 7204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:16:18.258988 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.258774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:60770)
0926 20:16:18.258862 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.258913 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 20:16:18.259076 6504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c5f033f13aa84455b5f010f47419e62a"
W20260926 20:16:18.259318 6665 leader_election.cc:334] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 797fee1bbcb9413db379bc44ca2f43ec (127.31.119.254:46179): Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
I20260926 20:16:18.259970 7204 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:16:18.261065 6504 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ebd8eaf5f5c4d87918f3b07d6c25a20 in term 2.
I20260926 20:16:18.261768 6665 leader_election.cc:302] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ebd8eaf5f5c4d87918f3b07d6c25a20, c5f033f13aa84455b5f010f47419e62a; no voters: 797fee1bbcb9413db379bc44ca2f43ec
I20260926 20:16:18.262033 7205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:16:18.262239 7205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [term 2 LEADER]: Becoming Leader. State: Replica: 4ebd8eaf5f5c4d87918f3b07d6c25a20, State: Running, Role: LEADER
I20260926 20:16:18.262516 7205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } }
I20260926 20:16:18.264153 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ebd8eaf5f5c4d87918f3b07d6c25a20. Latest consensus state: current_term: 2 leader_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:18.264295 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:18.264716 7216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:18.267089 7216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:18.267463 7216 catalog_manager.cc:1259] Loaded cluster ID: 14dc690dc703444da8ff268d5ac1b4de
I20260926 20:16:18.267513 7216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:18.268021 7216 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:18.268360 7216 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Loaded TSK: 0
I20260926 20:16:18.269011 7216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:16:18.296794 6679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:16:18.299962 6679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:16:18.303220 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.302946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:60792)
0926 20:16:18.303092 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.303145 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:16:18.303279 6504 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [term 2 FOLLOWER]: Refusing update from remote peer 4ebd8eaf5f5c4d87918f3b07d6c25a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 20:16:18.304035 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.303796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.0.0.1:60796)
0926 20:16:18.303921 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.303974 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:16:18.304117 7214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 20:16:18.304304 6665 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 -> Peer 797fee1bbcb9413db379bc44ca2f43ec (127.31.119.254:46179): Couldn't send request to peer 797fee1bbcb9413db379bc44ca2f43ec. Status: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 20:16:18.305933 7204 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ebd8eaf5f5c4d87918f3b07d6c25a20. Latest consensus state: current_term: 2 leader_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:18.306093 7204 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:16:18.306959 7217 mvcc.cc:205] Tried to move back new op lower bound from 7333698675924615168 to 7333698675765927936. Current Snapshot: MvccSnapshot[applied={T|T < 7333698675924615168}]
I20260926 20:16:18.307235 7205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:18.307368 7205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:18.307732 7218 mvcc.cc:205] Tried to move back new op lower bound from 7333698675924615168 to 7333698675765927936. Current Snapshot: MvccSnapshot[applied={T|T < 7333698675924615168}]
I20260926 20:16:18.307946 7204 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "797fee1bbcb9413db379bc44ca2f43ec" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 46179 } } peers { permanent_uuid: "c5f033f13aa84455b5f010f47419e62a" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 34189 } attrs { promote: false } } peers { permanent_uuid: "4ebd8eaf5f5c4d87918f3b07d6c25a20" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43255 } attrs { promote: false } } }
I20260926 20:16:18.308089 7204 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f033f13aa84455b5f010f47419e62a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:16:18.319044 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.314146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:42921 (local address 127.0.0.1:57630)
0926 20:16:18.314404 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.314421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:18.314797 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:18.315396 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:18.315399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:18.315414 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:18.315507 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.315512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.316553 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.316555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.317658 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.317667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.318656 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.318659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.318720 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.318736 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:16:18.318819 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:16:18.318896 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":59}
I20260926 20:16:18.319057 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.314382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.31.119.194:42921)
0926 20:16:18.315155 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:18.315160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:18.315175 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:18.315231 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:18.315234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:18.315292 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:18.315397 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:18.315637 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.316406 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.317799 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.318463 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.318535 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.318585 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:18.318847 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 20:16:18.318923 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":81,"threads_started":1}
I20260926 20:16:18.319300 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.314436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35571 (local address 127.0.0.1:55322)
0926 20:16:18.314617 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.314628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:18.314697 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:18.315265 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:18.315268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:18.315286 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:18.315392 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.315398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.316476 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.316480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.317842 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.317852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.318823 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.318827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.318945 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.318972 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:16:18.319027 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:16:18.319112 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":42}
I20260926 20:16:18.319692 6931 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae85a43c084cfca6ead7ef7e475612 (DEFAULT_TABLE table=test-table [id=89bfd5fd65e843959cc429a388947e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:18.319983 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.314726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.31.119.193:35571)
0926 20:16:18.314997 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:18.315003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:18.315019 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:16:18.315075 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:18.315080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:18.315127 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:18.315242 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:18.315518 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.316352 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.318009 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.318678 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.319620 (+ 942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.319682 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:18.319723 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 20:16:18.319811 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
I20260926 20:16:18.320037 6931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae85a43c084cfca6ead7ef7e475612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:18.320487 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.314960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.31.119.195:33481)
0926 20:16:18.315672 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:18.315675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:18.315688 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:16:18.315731 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:18.315734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:18.315772 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:18.315889 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:18.316188 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.316975 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.318369 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.319064 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.319676 (+ 612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.319732 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:18.320241 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 20:16:18.320331 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":69,"threads_started":1}
I20260926 20:16:18.320516 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.314825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:33481 (local address 127.0.0.1:59998)
0926 20:16:18.315371 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.315387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:18.315478 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:18.315898 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:18.315901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:18.315918 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:18.316046 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.316052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.317116 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.317119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.318239 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.318248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.319781 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.319784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.319901 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.319924 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:16:18.320016 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:16:18.320362 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"spinlock_wait_cycles":896,"thread_start_us":109,"threads_started":1}
I20260926 20:16:18.320508 6793 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae85a43c084cfca6ead7ef7e475612 (DEFAULT_TABLE table=test-table [id=89bfd5fd65e843959cc429a388947e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:18.320799 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae85a43c084cfca6ead7ef7e475612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:18.321218 7068 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae85a43c084cfca6ead7ef7e475612 (DEFAULT_TABLE table=test-table [id=89bfd5fd65e843959cc429a388947e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:18.321498 7068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae85a43c084cfca6ead7ef7e475612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:18.322824 7226 tablet_bootstrap.cc:492] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: Bootstrap starting.
I20260926 20:16:18.323310 7227 tablet_bootstrap.cc:492] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4: Bootstrap starting.
I20260926 20:16:18.323998 7226 tablet_bootstrap.cc:654] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:18.323995 7228 tablet_bootstrap.cc:492] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: Bootstrap starting.
I20260926 20:16:18.324193 7227 tablet_bootstrap.cc:654] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:18.324972 7228 tablet_bootstrap.cc:654] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:18.325564 7227 tablet_bootstrap.cc:492] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4: No bootstrap required, opened a new log
I20260926 20:16:18.325685 7227 ts_tablet_manager.cc:1397] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:16:18.325997 7226 tablet_bootstrap.cc:492] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: No bootstrap required, opened a new log
I20260926 20:16:18.326073 7226 ts_tablet_manager.cc:1397] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:18.326186 7227 raft_consensus.cc:348] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } }
I20260926 20:16:18.326332 7227 raft_consensus.cc:374] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:18.326380 7227 raft_consensus.cc:729] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4e11f6773e4724995df855fed7f7d4, State: Initialized, Role: FOLLOWER
I20260926 20:16:18.326371 7226 raft_consensus.cc:348] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } }
I20260926 20:16:18.326598 7226 raft_consensus.cc:374] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:18.326643 7226 raft_consensus.cc:729] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0866b3d3c243a1b9349584dbbc8355, State: Initialized, Role: FOLLOWER
I20260926 20:16:18.326676 7227 consensus_queue.cc:260] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } }
I20260926 20:16:18.326777 7226 consensus_queue.cc:260] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } }
I20260926 20:16:18.327111 7227 ts_tablet_manager.cc:1428] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:18.327147 7226 ts_tablet_manager.cc:1428] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:18.328280 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.327461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.194:36513)
0926 20:16:18.328108 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.328183 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":94,"threads_started":1}
I20260926 20:16:18.329181 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.328695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.194:41409)
0926 20:16:18.329046 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.329098 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268}
I20260926 20:16:18.329475 7228 tablet_bootstrap.cc:492] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: No bootstrap required, opened a new log
I20260926 20:16:18.329594 7228 ts_tablet_manager.cc:1397] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 20:16:18.330152 6859 heartbeater.cc:503] Master 127.31.119.252:43255 was elected leader, sending a full tablet report...
I20260926 20:16:18.330164 7228 raft_consensus.cc:348] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } }
I20260926 20:16:18.330266 7228 raft_consensus.cc:374] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:18.330297 7228 raft_consensus.cc:729] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dea1409891644e685da300de2183ef3, State: Initialized, Role: FOLLOWER
I20260926 20:16:18.330420 7228 consensus_queue.cc:260] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } }
I20260926 20:16:18.330778 7228 ts_tablet_manager.cc:1428] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:18.331637 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.331164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.195:60935)
0926 20:16:18.331453 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.331527 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":106,"threads_started":1}
I20260926 20:16:18.332794 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.327707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.193:40957)
0926 20:16:18.332624 (+ 4917us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.332691 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4837,"thread_start_us":87,"threads_started":1}
I20260926 20:16:18.333036 7134 heartbeater.cc:503] Master 127.31.119.252:43255 was elected leader, sending a full tablet report...
I20260926 20:16:18.333472 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.333191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.195:52807)
0926 20:16:18.333339 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.333393 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:16:18.333762 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.333263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.193:59185)
0926 20:16:18.333621 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.333678 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":126,"threads_started":1}
W20260926 20:16:18.348938 7137 tablet.cc:2369] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:16:18.374017 7000 tablet.cc:2369] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:16:18.392258 7233 raft_consensus.cc:482] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:16:18.392409 7233 raft_consensus.cc:504] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } }
I20260926 20:16:18.393841 7233 leader_election.cc:288] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4e11f6773e4724995df855fed7f7d4 (127.31.119.193:35571), 4dea1409891644e685da300de2183ef3 (127.31.119.195:33481)
I20260926 20:16:18.397876 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.394088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35571 (local address 127.31.119.194:42587)
0926 20:16:18.394223 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.394242 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:18.394344 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:18.394806 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:18.394810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:18.394831 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:18.394979 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.394987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.395774 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.395777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.396664 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.396674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.397411 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.397415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.397531 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.397550 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:16:18.397614 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 20:16:18.397693 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 20:16:18.397881 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.394347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:42587 (local address 127.31.119.193:35571)
0926 20:16:18.394545 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:18.394551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:18.394568 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:16:18.394624 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:18.394627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:18.394689 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:18.394769 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:18.395098 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.395657 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.396819 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.397233 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.397414 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.397472 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:18.397661 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:16:18.397747 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:16:18.397925 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.394088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:33481 (local address 127.31.119.194:58069)
0926 20:16:18.394629 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.394642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:18.394749 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:18.394986 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:18.394988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:18.395003 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:18.395125 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.395132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.395920 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.395922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.396674 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:18.396683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.397397 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.397401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:18.397507 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.397617 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 20:16:18.397701 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:16:18.397765 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":133,"thread_start_us":138,"threads_started":1}
I20260926 20:16:18.397941 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.394263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:58069 (local address 127.31.119.195:33481)
0926 20:16:18.394384 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:18.394389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:18.394756 (+ 367us) server_negotiation.cc:407] Connection header received
0926 20:16:18.394824 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:18.394827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:18.394874 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:18.394959 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:18.395240 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.395788 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.396818 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.397225 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.397411 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.397467 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:18.397740 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 20:16:18.397823 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:16:18.398383 6813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae85a43c084cfca6ead7ef7e475612" candidate_uuid: "9c0866b3d3c243a1b9349584dbbc8355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4e11f6773e4724995df855fed7f7d4" is_pre_election: true
I20260926 20:16:18.398430 7088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae85a43c084cfca6ead7ef7e475612" candidate_uuid: "9c0866b3d3c243a1b9349584dbbc8355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dea1409891644e685da300de2183ef3" is_pre_election: true
I20260926 20:16:18.398568 6813 raft_consensus.cc:2454] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c0866b3d3c243a1b9349584dbbc8355 in term 0.
I20260926 20:16:18.398639 7088 raft_consensus.cc:2454] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c0866b3d3c243a1b9349584dbbc8355 in term 0.
I20260926 20:16:18.398989 6885 leader_election.cc:302] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4e11f6773e4724995df855fed7f7d4, 9c0866b3d3c243a1b9349584dbbc8355; no voters:
I20260926 20:16:18.399245 7233 raft_consensus.cc:2781] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:16:18.399304 7233 raft_consensus.cc:482] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:16:18.399341 7233 raft_consensus.cc:3037] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:18.400347 7233 raft_consensus.cc:504] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } }
I20260926 20:16:18.400727 7233 leader_election.cc:288] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [CANDIDATE]: Term 1 election: Requested vote from peers 7d4e11f6773e4724995df855fed7f7d4 (127.31.119.193:35571), 4dea1409891644e685da300de2183ef3 (127.31.119.195:33481)
I20260926 20:16:18.401084 6813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae85a43c084cfca6ead7ef7e475612" candidate_uuid: "9c0866b3d3c243a1b9349584dbbc8355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4e11f6773e4724995df855fed7f7d4"
I20260926 20:16:18.401242 6813 raft_consensus.cc:3037] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:18.401233 7088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae85a43c084cfca6ead7ef7e475612" candidate_uuid: "9c0866b3d3c243a1b9349584dbbc8355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dea1409891644e685da300de2183ef3"
I20260926 20:16:18.401383 7088 raft_consensus.cc:3037] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:18.402537 6813 raft_consensus.cc:2454] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c0866b3d3c243a1b9349584dbbc8355 in term 1.
I20260926 20:16:18.402531 7088 raft_consensus.cc:2454] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c0866b3d3c243a1b9349584dbbc8355 in term 1.
I20260926 20:16:18.402835 6886 leader_election.cc:302] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dea1409891644e685da300de2183ef3, 9c0866b3d3c243a1b9349584dbbc8355; no voters:
I20260926 20:16:18.403041 7233 raft_consensus.cc:2781] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:18.403246 7233 raft_consensus.cc:686] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 1 LEADER]: Becoming Leader. State: Replica: 9c0866b3d3c243a1b9349584dbbc8355, State: Running, Role: LEADER
I20260926 20:16:18.403621 6997 heartbeater.cc:503] Master 127.31.119.252:43255 was elected leader, sending a full tablet report...
I20260926 20:16:18.403766 7233 consensus_queue.cc:237] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } } peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } }
I20260926 20:16:18.404093 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.403860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.194:51969)
0926 20:16:18.403959 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.404017 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 20:16:18.406755 6679 catalog_manager.cc:5495] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c0866b3d3c243a1b9349584dbbc8355 (127.31.119.194). New cstate: current_term: 1 leader_uuid: "9c0866b3d3c243a1b9349584dbbc8355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c0866b3d3c243a1b9349584dbbc8355" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 42921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 } health_report { overall_health: UNKNOWN } } }
W20260926 20:16:18.418828 6862 tablet.cc:2369] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:16:18.422166 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.418496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.31.119.194:42921)
0926 20:16:18.418713 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:18.418717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:18.418737 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:16:18.418808 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:18.418810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:18.418863 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:18.418989 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:18.419267 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.419784 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.420648 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:18.420872 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:18.420982 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:18.421253 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:18.421764 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:18.421897 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:18.421958 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 20:16:18.422016 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260926 20:16:18.425135 6813 raft_consensus.cc:1261] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4 [term 1 FOLLOWER]: Refusing update from remote peer 9c0866b3d3c243a1b9349584dbbc8355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:18.425364 7088 raft_consensus.cc:1261] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3 [term 1 FOLLOWER]: Refusing update from remote peer 9c0866b3d3c243a1b9349584dbbc8355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:18.425839 7240 consensus_queue.cc:1035] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4e11f6773e4724995df855fed7f7d4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:18.426093 7233 consensus_queue.cc:1035] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dea1409891644e685da300de2183ef3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 33481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:18.427990 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.427697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.195:46943)
0926 20:16:18.427835 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.427899 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 20:16:18.428283 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.427959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.193:56779)
0926 20:16:18.428130 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.428188 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 20:16:18.431118 7244 mvcc.cc:205] Tried to move back new op lower bound from 7333698676424065024 to 7333698676344770560. Current Snapshot: MvccSnapshot[applied={T|T < 7333698676424065024}]
I20260926 20:16:18.432793 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.432043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.194:60591)
0926 20:16:18.432639 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.432710 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519}
I20260926 20:16:18.434007 7245 mvcc.cc:205] Tried to move back new op lower bound from 7333698676424065024 to 7333698676344770560. Current Snapshot: MvccSnapshot[applied={T|T < 7333698676424065024}]
I20260926 20:16:18.434192 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.433624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.193:47237)
0926 20:16:18.434041 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.434113 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
I20260926 20:16:18.434787 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.433789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.194:48005)
0926 20:16:18.434667 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.434720 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":805}
I20260926 20:16:18.435765 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.435549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:46179 (local address 127.31.119.195:41135)
0926 20:16:18.435634 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.435689 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 20:16:18.442306 6678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:18.442509 6678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:18.446551 6678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebd8eaf5f5c4d87918f3b07d6c25a20: Sending DeleteTablet for 3 replicas of tablet 38ae85a43c084cfca6ead7ef7e475612
I20260926 20:16:18.447359 6931 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae85a43c084cfca6ead7ef7e475612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:18 UTC) from {username='oryx', principal='oryx/127.31.119.252@KRBTEST.COM'} at 127.0.0.1:57630
I20260926 20:16:18.447350 7068 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae85a43c084cfca6ead7ef7e475612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:18 UTC) from {username='oryx', principal='oryx/127.31.119.252@KRBTEST.COM'} at 127.0.0.1:59998
I20260926 20:16:18.447573 6793 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae85a43c084cfca6ead7ef7e475612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:18 UTC) from {username='oryx', principal='oryx/127.31.119.252@KRBTEST.COM'} at 127.0.0.1:55322
I20260926 20:16:18.447948 7255 tablet_replica.cc:330] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4: stopping tablet replica
I20260926 20:16:18.448102 7255 raft_consensus.cc:2229] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:18.448249 7255 raft_consensus.cc:2258] T 38ae85a43c084cfca6ead7ef7e475612 P 7d4e11f6773e4724995df855fed7f7d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:18.448506 7253 tablet_replica.cc:330] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: stopping tablet replica
I20260926 20:16:18.448701 7253 raft_consensus.cc:2229] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:16:18.448814 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 6641
I20260926 20:16:18.450851 7253 raft_consensus.cc:2258] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:18.451524 7253 ts_tablet_manager.cc:1905] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:18.453728 7253 ts_tablet_manager.cc:1918] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:18.453828 7253 log.cc:1196] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/38ae85a43c084cfca6ead7ef7e475612
I20260926 20:16:18.454097 7253 ts_tablet_manager.cc:1939] T 38ae85a43c084cfca6ead7ef7e475612 P 9c0866b3d3c243a1b9349584dbbc8355: Deleting consensus metadata
I20260926 20:16:18.454943 7254 tablet_replica.cc:330] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: stopping tablet replica
I20260926 20:16:18.454978 6663 catalog_manager.cc:4841] TS 9c0866b3d3c243a1b9349584dbbc8355 (127.31.119.194:42921): tablet 38ae85a43c084cfca6ead7ef7e475612 (table test-table [id=89bfd5fd65e843959cc429a388947e11]) successfully deleted
I20260926 20:16:18.455084 7254 raft_consensus.cc:2229] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:18.455222 7254 raft_consensus.cc:2258] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:18.456065 7254 ts_tablet_manager.cc:1905] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:18.456933 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 6867
I20260926 20:16:18.457158 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:18.456880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35571 (local address 127.0.0.1:55330)
0926 20:16:18.456989 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:18.457058 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:35571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 20:16:18.458968 6665 catalog_manager.cc:4584] TS 7d4e11f6773e4724995df855fed7f7d4 (127.31.119.193:35571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38ae85a43c084cfca6ead7ef7e475612: Network error: Client connection negotiation failed: client connection to 127.31.119.193:35571: connect: Connection refused (error 111)
I20260926 20:16:18.460183 7254 ts_tablet_manager.cc:1918] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:18.460254 7254 log.cc:1196] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/38ae85a43c084cfca6ead7ef7e475612
I20260926 20:16:18.460556 7254 ts_tablet_manager.cc:1939] T 38ae85a43c084cfca6ead7ef7e475612 P 4dea1409891644e685da300de2183ef3: Deleting consensus metadata
I20260926 20:16:18.461359 6666 catalog_manager.cc:4841] TS 4dea1409891644e685da300de2183ef3 (127.31.119.195:33481): tablet 38ae85a43c084cfca6ead7ef7e475612 (table test-table [id=89bfd5fd65e843959cc429a388947e11]) successfully deleted
I20260926 20:16:18.463876 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 7004
I20260926 20:16:18.471148 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 6356
I20260926 20:16:18.477188 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 6461
2026-09-26T20:16:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5446 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:16:18 dist-test-slave-vnbh krb5kdc[7258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:16:18 dist-test-slave-vnbh krb5kdc[7258](info): set up 2 sockets
Sep 26 20:16:18 dist-test-slave-vnbh krb5kdc[7258](info): commencing operation
krb5kdc: starting...
W20260926 20:16:19.531015 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:19 dist-test-slave-vnbh krb5kdc[7258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.254.keytab.
Entry for principal oryx/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.254.keytab.
I20260926 20:16:19.665161 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:36527
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.254.keytab
--principal=oryx/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:19.781380 7271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:19.781685 7271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:19.781729 7271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:19.785874 7271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:19.785943 7271 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:16:19.785960 7271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:19.785979 7271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:19.785995 7271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:19.786016 7271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:19.791790 7271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:36527
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.254.keytab
--principal=oryx/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:19.793025 7271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:19.794246 7271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:19.804512 7276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:19.804553 7279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:19.804733 7277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:19.804912 7271 server_base.cc:1034] running on GCE node
I20260926 20:16:19.805739 7271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:19.807005 7271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:19.808205 7271 hybrid_clock.cc:648] HybridClock initialized: now 1790453779808193 us; error 34 us; skew 500 ppm
Sep 26 20:16:19 dist-test-slave-vnbh krb5kdc[7258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453779, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:19.811203 7271 init.cc:377] Logged in from keytab as oryx/127.31.119.254@KRBTEST.COM (short username oryx)
I20260926 20:16:19.812180 7271 webserver.cc:460] Webserver started at http://127.31.119.254:42415/ using document root <none> and password file <none>
I20260926 20:16:19.812812 7271 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:19.812881 7271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:19.813117 7271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:19.815303 7271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f"
format_stamp: "Formatted at 2026-09-26 20:16:19 on dist-test-slave-vnbh"
server_key: "dad56146580d3689f99cf7363cdfb721"
server_key_iv: "1c6224d11238582b3b3dfb7b37c26b1f"
server_key_version: "encryptionkey@0"
I20260926 20:16:19.815938 7271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f"
format_stamp: "Formatted at 2026-09-26 20:16:19 on dist-test-slave-vnbh"
server_key: "dad56146580d3689f99cf7363cdfb721"
server_key_iv: "1c6224d11238582b3b3dfb7b37c26b1f"
server_key_version: "encryptionkey@0"
I20260926 20:16:19.820025 7271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 20:16:19.822738 7286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:19.824080 7271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:16:19.824216 7271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f"
format_stamp: "Formatted at 2026-09-26 20:16:19 on dist-test-slave-vnbh"
server_key: "dad56146580d3689f99cf7363cdfb721"
server_key_iv: "1c6224d11238582b3b3dfb7b37c26b1f"
server_key_version: "encryptionkey@0"
I20260926 20:16:19.824339 7271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:19.851080 7271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:19.854943 7271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:19.855170 7271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:19.863799 7271 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:36527
I20260926 20:16:19.863814 7348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:36527 every 8 connection(s)
I20260926 20:16:19.864864 7271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:19.867977 7349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:19.872589 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 7271
I20260926 20:16:19.872718 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:19.873087 32223 external_mini_cluster.cc:1428] Setting key f0ff4b6c72271ca3d3b6dd1c16f59d0b
I20260926 20:16:19.874524 7349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f: Bootstrap starting.
I20260926 20:16:19.877100 7349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:19.878103 7349 log.cc:824] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:19.880666 7349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f: No bootstrap required, opened a new log
Sep 26 20:16:19 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:19.884760 7349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36527 } }
I20260926 20:16:19.885099 7349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:19.885188 7349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3caba2ac7a4cfd94b3b07c0008d70f, State: Initialized, Role: FOLLOWER
I20260926 20:16:19.885772 7349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36527 } }
I20260926 20:16:19.885938 7349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:19.886005 7349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:19.886104 7349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:19.887091 7349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36527 } }
I20260926 20:16:19.887460 7349 leader_election.cc:302] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c3caba2ac7a4cfd94b3b07c0008d70f; no voters:
I20260926 20:16:19.887874 7349 leader_election.cc:288] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:19.888015 7354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:19.888247 7354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [term 1 LEADER]: Becoming Leader. State: Replica: 6c3caba2ac7a4cfd94b3b07c0008d70f, State: Running, Role: LEADER
I20260926 20:16:19.888612 7354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36527 } }
I20260926 20:16:19.889266 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:19.874744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.31.119.254:36527)
0926 20:16:19.875204 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:19.875230 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:19.875267 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:16:19.875969 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:19.875987 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:19.876363 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:19.876693 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:19.877777 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:19.879058 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:19.880258 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:19.880789 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:19.880971 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:19.883720 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:19.883725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:19.883741 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:19.883774 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:19.885851 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:19.886517 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:19.886524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:19.886537 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:19.886660 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:19.886970 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:19.886976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:19.886978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:19.887458 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:19.887672 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:19.887997 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 20:16:19.888346 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":172,"threads_started":1}
I20260926 20:16:19.889353 7349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:19.889923 7354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3caba2ac7a4cfd94b3b07c0008d70f. Latest consensus state: current_term: 1 leader_uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36527 } } }
I20260926 20:16:19.890059 7354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:19.890496 7355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3caba2ac7a4cfd94b3b07c0008d70f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 36527 } } }
I20260926 20:16:19.890656 7355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f [sys.catalog]: This master's current role is: LEADER
W20260926 20:16:19.894615 7370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:16:19.894733 7370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:16:19.894843 7359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:19.895643 7359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:19.902096 7359 catalog_manager.cc:1347] Generated new cluster ID: 3596a17deadb481f9b3d0325844c4fc9
I20260926 20:16:19.902186 7359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:19.928443 7359 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:19.929762 7359 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:19.936882 7359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f: Generated new TSK 0
I20260926 20:16:19.937809 7359 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab.
Entry for principal oryx/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab.
I20260926 20:16:19.978130 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:36527
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab
--principal=oryx/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:20.095031 7376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:20.095329 7376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:20.095376 7376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:20.099861 7376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:20.099931 7376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:20.099973 7376 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:16:20.100013 7376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:20.105819 7376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab
--principal=oryx/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.119.254:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:20.107087 7376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:20.108189 7376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:20.119187 7381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:20.119172 7382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:20.119172 7384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:20.119419 7376 server_base.cc:1034] running on GCE node
I20260926 20:16:20.119855 7376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:20.120615 7376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:20.121831 7376 hybrid_clock.cc:648] HybridClock initialized: now 1790453780121808 us; error 34 us; skew 500 ppm
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453780, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:20.124984 7376 init.cc:377] Logged in from keytab as oryx/127.31.119.193@KRBTEST.COM (short username oryx)
I20260926 20:16:20.125999 7376 webserver.cc:460] Webserver started at http://127.31.119.193:43763/ using document root <none> and password file <none>
I20260926 20:16:20.126555 7376 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:20.126617 7376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:20.126785 7376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:20.129160 7376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "1e5af0faa4e4465d8b19b495fc7cd943"
format_stamp: "Formatted at 2026-09-26 20:16:20 on dist-test-slave-vnbh"
server_key: "8a448f0708bc406152940eac12937a65"
server_key_iv: "0cb4029fbe9c60ce7a31eee1f15917a9"
server_key_version: "encryptionkey@0"
I20260926 20:16:20.129884 7376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "1e5af0faa4e4465d8b19b495fc7cd943"
format_stamp: "Formatted at 2026-09-26 20:16:20 on dist-test-slave-vnbh"
server_key: "8a448f0708bc406152940eac12937a65"
server_key_iv: "0cb4029fbe9c60ce7a31eee1f15917a9"
server_key_version: "encryptionkey@0"
I20260926 20:16:20.134061 7376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:20.136788 7391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:20.138352 7376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:20.138540 7376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "1e5af0faa4e4465d8b19b495fc7cd943"
format_stamp: "Formatted at 2026-09-26 20:16:20 on dist-test-slave-vnbh"
server_key: "8a448f0708bc406152940eac12937a65"
server_key_iv: "0cb4029fbe9c60ce7a31eee1f15917a9"
server_key_version: "encryptionkey@0"
I20260926 20:16:20.138687 7376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:20.165328 7376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:20.168903 7376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:20.169224 7376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:20.170769 7376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:20.170820 7376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:20.170861 7376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:20.170923 7376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453780, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:20.182577 7376 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:36937
I20260926 20:16:20.182628 7507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:36937 every 8 connection(s)
I20260926 20:16:20.183810 7376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:20.185494 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 7376
I20260926 20:16:20.185633 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:16:20.185988 32223 external_mini_cluster.cc:1428] Setting key a06ea52d22966a4b78be248638b9504f
I20260926 20:16:20.186647 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.172056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:43673 (local address 127.31.119.254:36527)
0926 20:16:20.172226 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.172232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.173146 (+ 914us) server_negotiation.cc:407] Connection header received
0926 20:16:20.174522 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.174527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.174598 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.174714 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:20.176359 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.176954 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.178638 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.178943 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.179083 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.182265 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:20.182270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:20.182273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:20.182311 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:20.184023 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.184726 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.184730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.184732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.184823 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.185205 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.185210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.185215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.185626 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:20.185736 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.186326 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.186472 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:16:20.187705 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.172457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36527 (local address 127.31.119.193:43673)
0926 20:16:20.172965 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.173024 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.174299 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.174914 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.174924 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.175492 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:20.176175 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.176191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.177105 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.177109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.178497 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.178508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.179128 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.179132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.179350 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.180082 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:20.180101 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:20.182035 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:20.184212 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.184221 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.184238 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.184591 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.184937 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.184939 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.184941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.185076 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.185745 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:20.185758 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:20.186009 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.186221 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.186664 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":144,"threads_started":1}
I20260926 20:16:20.189339 7508 heartbeater.cc:347] Connected to a master server at 127.31.119.254:36527
I20260926 20:16:20.189497 7508 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:20.190196 7508 heartbeater.cc:511] Master 127.31.119.254:36527 requested a full tablet report, sending...
I20260926 20:16:20.192250 7303 ts_manager.cc:194] Registered new tserver with Master: 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937)
I20260926 20:16:20.193810 7303 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:43673
WARNING: no policy specified for oryx/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.194@KRBTEST.COM" created.
I20260926 20:16:20.202226 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.192384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:59287 (local address 127.31.119.254:36527)
0926 20:16:20.192511 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.192515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.193027 (+ 512us) server_negotiation.cc:407] Connection header received
0926 20:16:20.193140 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.193143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.193191 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.193304 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:20.194009 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.194979 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.196524 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.196753 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.196845 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.198425 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:20.198428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:20.198430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:20.198453 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:20.200058 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.200546 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.200550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.200551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.200680 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.201148 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.201151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.201152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.201511 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:20.201641 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.201924 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.202041 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:16:20.202240 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.192312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36527 (local address 127.31.119.193:59287)
0926 20:16:20.192908 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.192927 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.193048 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.193351 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.193355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.193646 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:20.193828 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.193841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.195176 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.195182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.196358 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.196374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.196902 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.196908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.197043 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.197693 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:20.197710 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:20.198261 (+ 551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:20.200184 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.200187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.200188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.200394 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.200824 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.200826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.200828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.200909 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.201636 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:20.201639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:20.201749 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.201859 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.202038 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab.
Entry for principal oryx/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab.
I20260926 20:16:20.220459 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:36527
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab
--principal=oryx/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:20.337616 7518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:20.337921 7518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:20.338001 7518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:20.342177 7518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:20.342268 7518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:20.342342 7518 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:16:20.342432 7518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:20.348106 7518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab
--principal=oryx/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.119.254:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:20.349458 7518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:20.350517 7518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:20.360353 7526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:20.360455 7524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:20.360456 7523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:20.360811 7518 server_base.cc:1034] running on GCE node
I20260926 20:16:20.361260 7518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:20.362098 7518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:20.363281 7518 hybrid_clock.cc:648] HybridClock initialized: now 1790453780363237 us; error 58 us; skew 500 ppm
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453780, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:20.366389 7518 init.cc:377] Logged in from keytab as oryx/127.31.119.194@KRBTEST.COM (short username oryx)
I20260926 20:16:20.367380 7518 webserver.cc:460] Webserver started at http://127.31.119.194:40839/ using document root <none> and password file <none>
I20260926 20:16:20.367975 7518 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:20.368045 7518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:20.368263 7518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:20.370472 7518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "555a74f1a076438db59c7dc87eb72db1"
format_stamp: "Formatted at 2026-09-26 20:16:20 on dist-test-slave-vnbh"
server_key: "ecabdb86774be64aeaff3bb5cec1c6c1"
server_key_iv: "0a24b43223c9c8fba37a8c573bb8161b"
server_key_version: "encryptionkey@0"
I20260926 20:16:20.371037 7518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "555a74f1a076438db59c7dc87eb72db1"
format_stamp: "Formatted at 2026-09-26 20:16:20 on dist-test-slave-vnbh"
server_key: "ecabdb86774be64aeaff3bb5cec1c6c1"
server_key_iv: "0a24b43223c9c8fba37a8c573bb8161b"
server_key_version: "encryptionkey@0"
I20260926 20:16:20.374912 7518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:20.377626 7533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:20.379017 7518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:20.379119 7518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "555a74f1a076438db59c7dc87eb72db1"
format_stamp: "Formatted at 2026-09-26 20:16:20 on dist-test-slave-vnbh"
server_key: "ecabdb86774be64aeaff3bb5cec1c6c1"
server_key_iv: "0a24b43223c9c8fba37a8c573bb8161b"
server_key_version: "encryptionkey@0"
I20260926 20:16:20.379267 7518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:20.398931 7518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:20.402804 7518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:20.403038 7518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:20.404614 7518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:20.404670 7518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:20.404744 7518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:20.404778 7518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453780, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:20.416374 7518 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:46023
I20260926 20:16:20.416455 7648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:46023 every 8 connection(s)
I20260926 20:16:20.417640 7518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:20.420110 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.405862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:36505 (local address 127.31.119.254:36527)
0926 20:16:20.406024 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.406027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.407682 (+ 1655us) server_negotiation.cc:407] Connection header received
0926 20:16:20.409137 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.409141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.409207 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.409327 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:20.410672 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.411340 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.412670 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.412987 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.413101 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.416257 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:20.416261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:20.416263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:20.416293 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:20.418071 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.418693 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.418696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.418698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.418797 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.419155 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.419158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.419159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.419347 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:20.419426 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.419882 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.419987 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:16:20.420960 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.406470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36527 (local address 127.31.119.194:36505)
0926 20:16:20.407329 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.407451 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.408876 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.409617 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.409625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.409986 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:20.410505 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.410520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.411478 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.411482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.412525 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.412536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.413151 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.413166 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.413410 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.414175 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:20.414194 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:20.415984 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:20.418204 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.418209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.418226 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.418560 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.418893 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.418895 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.418896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.419066 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.419437 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:20.419442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:20.419674 (+ 232us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.419832 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.420124 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":155,"threads_started":1}
I20260926 20:16:20.422544 7649 heartbeater.cc:347] Connected to a master server at 127.31.119.254:36527
I20260926 20:16:20.422770 7649 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:20.423352 7649 heartbeater.cc:511] Master 127.31.119.254:36527 requested a full tablet report, sending...
I20260926 20:16:20.424506 7299 ts_manager.cc:194] Registered new tserver with Master: 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023)
I20260926 20:16:20.425419 7299 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:36505
I20260926 20:16:20.427748 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 7518
I20260926 20:16:20.427891 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:20.428234 32223 external_mini_cluster.cc:1428] Setting key c681f1ac5d61cc60c0d5119fe4ebeceb
I20260926 20:16:20.432739 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.424631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:45729 (local address 127.31.119.254:36527)
0926 20:16:20.424745 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.424748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.424917 (+ 169us) server_negotiation.cc:407] Connection header received
0926 20:16:20.424989 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.424992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.425035 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.425123 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:20.425702 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.426398 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.427237 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.427556 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.427639 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.428974 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:20.428982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:20.428985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:20.429016 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:20.430750 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.431371 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.431374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.431375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.431457 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.431835 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.431840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.431842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.432091 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:20.432203 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.432463 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.432578 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:16:20.432750 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.424547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36527 (local address 127.31.119.194:45729)
0926 20:16:20.424828 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.424840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.424921 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.425144 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.425146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.425462 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:20.425613 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.425619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.426550 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.426552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.427121 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.427129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.427678 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.427683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.427816 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.428388 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:20.428411 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:20.428768 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:20.430917 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.430922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.430928 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.431262 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.431636 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.431639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.431641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.431738 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.432193 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:20.432196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:20.432292 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.432399 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.432564 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for oryx/127.31.119.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab.
Entry for principal oryx/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab.
I20260926 20:16:20.457173 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:36527
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab
--principal=oryx/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:20.575323 7658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:20.575621 7658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:20.575703 7658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:20.579937 7658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:20.580011 7658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:20.580053 7658 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:16:20.580147 7658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:20.586336 7658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab
--principal=oryx/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.119.254:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:20.587548 7658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:20.588641 7658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:20.599721 7666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:20.599713 7664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:20.599720 7663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:20.600204 7658 server_base.cc:1034] running on GCE node
I20260926 20:16:20.600708 7658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:20.601475 7658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:20.602677 7658 hybrid_clock.cc:648] HybridClock initialized: now 1790453780602657 us; error 36 us; skew 500 ppm
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453780, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:20.605762 7658 init.cc:377] Logged in from keytab as oryx/127.31.119.195@KRBTEST.COM (short username oryx)
I20260926 20:16:20.606722 7658 webserver.cc:460] Webserver started at http://127.31.119.195:36873/ using document root <none> and password file <none>
I20260926 20:16:20.607307 7658 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:20.607372 7658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:20.607553 7658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:20.609819 7658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "78efdddf1aff4afe9d5c59f5c9e36672"
format_stamp: "Formatted at 2026-09-26 20:16:20 on dist-test-slave-vnbh"
server_key: "026dea9150c0309ea58f908cf73797d3"
server_key_iv: "fc3a59e2a5891ff4c85c62acc7affec8"
server_key_version: "encryptionkey@0"
I20260926 20:16:20.610370 7658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "78efdddf1aff4afe9d5c59f5c9e36672"
format_stamp: "Formatted at 2026-09-26 20:16:20 on dist-test-slave-vnbh"
server_key: "026dea9150c0309ea58f908cf73797d3"
server_key_iv: "fc3a59e2a5891ff4c85c62acc7affec8"
server_key_version: "encryptionkey@0"
I20260926 20:16:20.614707 7658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:16:20.617461 7674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:20.618940 7658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:20.619063 7658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "78efdddf1aff4afe9d5c59f5c9e36672"
format_stamp: "Formatted at 2026-09-26 20:16:20 on dist-test-slave-vnbh"
server_key: "026dea9150c0309ea58f908cf73797d3"
server_key_iv: "fc3a59e2a5891ff4c85c62acc7affec8"
server_key_version: "encryptionkey@0"
I20260926 20:16:20.619166 7658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:20.637431 7658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:20.641822 7658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:20.642002 7658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:20.643879 7658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:20.643962 7658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:20.644009 7658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:20.644032 7658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453780, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:20.655144 7658 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:37457
I20260926 20:16:20.655230 7789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:37457 every 8 connection(s)
I20260926 20:16:20.656185 7658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:20.659515 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.644934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:51267 (local address 127.31.119.254:36527)
0926 20:16:20.645100 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.645108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.646184 (+ 1076us) server_negotiation.cc:407] Connection header received
0926 20:16:20.647631 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.647636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.647701 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.647793 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:20.649600 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.650555 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.651760 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.652008 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.652137 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.655440 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:20.655444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:20.655447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:20.655477 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:20.657246 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.657925 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.657928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.657929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.658018 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.658331 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.658333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.658334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.658633 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:20.658715 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.659251 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.659373 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:16:20.660338 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.645409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36527 (local address 127.31.119.195:51267)
0926 20:16:20.645980 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.646045 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.647319 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.648029 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.648039 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.648608 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:20.649437 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.649460 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.650691 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.650694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.651623 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.651633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.652173 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.652177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.652377 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.653153 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:20.653174 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:20.655227 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:20.657429 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.657434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.657447 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.657796 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.658103 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.658105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.658107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.658243 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.658728 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:20.658734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:20.658975 (+ 241us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.659131 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.659402 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":119,"threads_started":1}
I20260926 20:16:20.661700 7790 heartbeater.cc:347] Connected to a master server at 127.31.119.254:36527
I20260926 20:16:20.661859 7790 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:20.662474 7790 heartbeater.cc:511] Master 127.31.119.254:36527 requested a full tablet report, sending...
I20260926 20:16:20.663717 7303 ts_manager.cc:194] Registered new tserver with Master: 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457)
I20260926 20:16:20.664502 7303 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:51267
I20260926 20:16:20.665180 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 7658
I20260926 20:16:20.665343 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:20.665729 32223 external_mini_cluster.cc:1428] Setting key 2847c0bb7aea1ab48fa5baa6dd1dbdf9
I20260926 20:16:20.667893 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:16:20.674346 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.664228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:39683 (local address 127.31.119.254:36527)
0926 20:16:20.664359 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.664364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.664589 (+ 225us) server_negotiation.cc:407] Connection header received
0926 20:16:20.664746 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.664752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.664796 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.664860 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:20.665773 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.666521 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.667836 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.668188 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.668273 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.670773 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:20.670777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:20.670786 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:20.670811 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:20.672405 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.672973 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.672980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.672985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.673073 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.673417 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.673422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.673424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.673632 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:20.673751 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.674041 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.674166 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:16:20.674361 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.664102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36527 (local address 127.31.119.195:39683)
0926 20:16:20.664500 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.664513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.664635 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.665125 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.665127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.665496 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:20.665646 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.665653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.666927 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.666930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.667708 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.667719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.669457 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.669461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.669613 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.670260 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:20.670273 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:20.670625 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:20.672628 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.672633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.672636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.672862 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.673181 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.673184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.673186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.673330 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.673740 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:20.673744 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:20.673853 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.673970 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.674141 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453780, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:16:20.694356 7352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:20.687376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.31.119.254:36527)
0926 20:16:20.687527 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.687531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.687542 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:16:20.687584 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.687586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.687625 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.687692 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:20.688424 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.688964 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.690109 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.690468 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.690610 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.694173 (+ 3563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56554: BlockingRecv error: recv got EOF from 127.0.0.1:56554 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453780, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:20.704499 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.695955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.31.119.254:36527)
0926 20:16:20.696111 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.696115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.696219 (+ 104us) server_negotiation.cc:407] Connection header received
0926 20:16:20.696374 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.696378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.696428 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.696512 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:20.697144 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.697752 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.698565 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.698801 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.698910 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.701362 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:20.701366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:20.701369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:20.701401 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:20.702754 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.703209 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.703213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.703214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.703301 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.703641 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.703644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.703646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.703838 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:20.703976 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.704217 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.704336 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:16:20.707849 7303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:16:20.711052 7303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:16:20.726490 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.719927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.31.119.194:46023)
0926 20:16:20.721090 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.721132 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.721158 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:16:20.721244 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.721253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.721553 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.721712 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:20.722042 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.723182 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.724789 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.725637 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.725793 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.725940 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.726248 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.726361 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":148,"threads_started":1}
I20260926 20:16:20.726856 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.719750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:46023 (local address 127.0.0.1:51410)
0926 20:16:20.720530 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.720585 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.720723 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.721721 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.721728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.721749 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:20.721917 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.721925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.723313 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.723316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.724651 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.724661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.725774 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.725777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.725887 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.725935 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.726045 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.726720 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":505,"thread_start_us":89,"threads_started":1}
I20260926 20:16:20.727140 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.720560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.31.119.195:37457)
0926 20:16:20.720987 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.721056 (+ 69us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.721136 (+ 80us) server_negotiation.cc:407] Connection header received
0926 20:16:20.721377 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.721383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.721552 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.721702 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:20.722454 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.723587 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.725228 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.726181 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.726335 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.726475 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.726851 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.726972 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":103,"threads_started":1}
I20260926 20:16:20.727607 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.720908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.31.119.193:36937)
0926 20:16:20.721309 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.721315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.721415 (+ 100us) server_negotiation.cc:407] Connection header received
0926 20:16:20.721542 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.721547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.721748 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.721937 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:20.722256 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.723530 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.725235 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.725946 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.726084 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.726223 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.727377 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.727477 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":99,"threads_started":1}
I20260926 20:16:20.727626 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.720491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:36937 (local address 127.0.0.1:48490)
0926 20:16:20.721312 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.721327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.721430 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.721914 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.721917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.721979 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:20.722117 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.722124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.723667 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.723672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.725091 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.725102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.726071 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.726075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.727225 (+ 1150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.727252 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.727348 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.727427 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":144,"thread_start_us":85,"threads_started":1}
I20260926 20:16:20.727626 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.719882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:37457 (local address 127.0.0.1:42004)
0926 20:16:20.721026 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.721040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.721162 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.721914 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.721917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.721976 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:20.722334 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.722341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.723719 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.723722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.725091 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.725102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.726546 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.726550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.726654 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.726699 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.727338 (+ 639us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.727427 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":323,"spinlock_wait_cycles":69376,"thread_start_us":77,"threads_started":1}
I20260926 20:16:20.730402 7724 tablet_service.cc:1467] Processing CreateTablet for tablet 32ba4486e7e6450499b95874f71290d1 (DEFAULT_TABLE table=test-table [id=b3c0e15bc2254ad29bd3d626f68a3c88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:20.730402 7442 tablet_service.cc:1467] Processing CreateTablet for tablet 32ba4486e7e6450499b95874f71290d1 (DEFAULT_TABLE table=test-table [id=b3c0e15bc2254ad29bd3d626f68a3c88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:20.730276 7583 tablet_service.cc:1467] Processing CreateTablet for tablet 32ba4486e7e6450499b95874f71290d1 (DEFAULT_TABLE table=test-table [id=b3c0e15bc2254ad29bd3d626f68a3c88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:20.731500 7442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ba4486e7e6450499b95874f71290d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:20.731501 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ba4486e7e6450499b95874f71290d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:20.731518 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ba4486e7e6450499b95874f71290d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:20.738780 7819 tablet_bootstrap.cc:492] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: Bootstrap starting.
I20260926 20:16:20.739261 7820 tablet_bootstrap.cc:492] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: Bootstrap starting.
I20260926 20:16:20.739776 7821 tablet_bootstrap.cc:492] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: Bootstrap starting.
I20260926 20:16:20.741783 7821 tablet_bootstrap.cc:654] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:20.741952 7819 tablet_bootstrap.cc:654] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:20.742686 7821 log.cc:824] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:20.742908 7819 log.cc:824] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:20.743324 7820 tablet_bootstrap.cc:654] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:20.744259 7820 log.cc:824] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:20.745030 7821 tablet_bootstrap.cc:492] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: No bootstrap required, opened a new log
I20260926 20:16:20.745035 7819 tablet_bootstrap.cc:492] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: No bootstrap required, opened a new log
I20260926 20:16:20.745249 7821 ts_tablet_manager.cc:1397] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:16:20.745415 7819 ts_tablet_manager.cc:1397] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 20:16:20.746443 7820 tablet_bootstrap.cc:492] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: No bootstrap required, opened a new log
I20260926 20:16:20.746662 7820 ts_tablet_manager.cc:1397] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 20:16:20.748661 7821 raft_consensus.cc:348] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } }
I20260926 20:16:20.748966 7821 raft_consensus.cc:374] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:20.749044 7821 raft_consensus.cc:729] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555a74f1a076438db59c7dc87eb72db1, State: Initialized, Role: FOLLOWER
I20260926 20:16:20.749571 7821 consensus_queue.cc:260] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } }
I20260926 20:16:20.750301 7649 heartbeater.cc:503] Master 127.31.119.254:36527 was elected leader, sending a full tablet report...
I20260926 20:16:20.750747 7821 ts_tablet_manager.cc:1428] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260926 20:16:20.750715 7819 raft_consensus.cc:348] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } }
I20260926 20:16:20.751003 7819 raft_consensus.cc:374] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:20.751055 7819 raft_consensus.cc:729] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5af0faa4e4465d8b19b495fc7cd943, State: Initialized, Role: FOLLOWER
I20260926 20:16:20.751195 7820 raft_consensus.cc:348] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } }
I20260926 20:16:20.751466 7820 raft_consensus.cc:374] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:20.751528 7820 raft_consensus.cc:729] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78efdddf1aff4afe9d5c59f5c9e36672, State: Initialized, Role: FOLLOWER
I20260926 20:16:20.751607 7819 consensus_queue.cc:260] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } }
I20260926 20:16:20.751979 7820 consensus_queue.cc:260] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } }
I20260926 20:16:20.752698 7819 ts_tablet_manager.cc:1428] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:16:20.752919 7820 ts_tablet_manager.cc:1428] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260926 20:16:20.753307 7790 heartbeater.cc:503] Master 127.31.119.254:36527 was elected leader, sending a full tablet report...
I20260926 20:16:20.753331 7508 heartbeater.cc:503] Master 127.31.119.254:36527 was elected leader, sending a full tablet report...
I20260926 20:16:20.782104 7825 raft_consensus.cc:482] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:16:20.782330 7825 raft_consensus.cc:504] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } }
I20260926 20:16:20.783546 7825 leader_election.cc:288] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457), 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937)
I20260926 20:16:20.788199 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.783971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:53091 (local address 127.31.119.193:36937)
0926 20:16:20.784106 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.784111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.784130 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:16:20.784220 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.784223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.784277 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.784370 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:20.784710 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.785608 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.786974 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.787595 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.787725 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.787795 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.788024 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.788097 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:16:20.788206 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.783839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:36937 (local address 127.31.119.194:53091)
0926 20:16:20.783980 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.783997 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.784090 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.784391 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.784393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.784410 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:20.784586 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.784594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.785720 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.785724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.786817 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.786825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.787726 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.787729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.787860 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.787886 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.787970 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.788046 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 20:16:20.788901 7462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ba4486e7e6450499b95874f71290d1" candidate_uuid: "555a74f1a076438db59c7dc87eb72db1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" is_pre_election: true
I20260926 20:16:20.789037 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.783975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:52041 (local address 127.31.119.195:37457)
0926 20:16:20.784106 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.784111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.784493 (+ 382us) server_negotiation.cc:407] Connection header received
0926 20:16:20.784612 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.784617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.784680 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.784756 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:20.785627 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.786437 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.787849 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.788470 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.788597 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.788648 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.788852 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.788921 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:16:20.789048 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.783839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:37457 (local address 127.31.119.194:52041)
0926 20:16:20.784378 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.784393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.784501 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.784781 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.784784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.784813 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:20.785480 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.785491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.786572 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.786574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.787724 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.787733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.788629 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.788633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.788730 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.788746 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.788837 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.788902 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":124,"thread_start_us":158,"threads_started":1}
I20260926 20:16:20.789322 7462 raft_consensus.cc:2454] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 555a74f1a076438db59c7dc87eb72db1 in term 0.
I20260926 20:16:20.789767 7744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ba4486e7e6450499b95874f71290d1" candidate_uuid: "555a74f1a076438db59c7dc87eb72db1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" is_pre_election: true
I20260926 20:16:20.790027 7537 leader_election.cc:302] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5af0faa4e4465d8b19b495fc7cd943, 555a74f1a076438db59c7dc87eb72db1; no voters:
I20260926 20:16:20.790081 7744 raft_consensus.cc:2454] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 555a74f1a076438db59c7dc87eb72db1 in term 0.
I20260926 20:16:20.790328 7825 raft_consensus.cc:2781] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:16:20.790468 7825 raft_consensus.cc:482] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:16:20.790534 7825 raft_consensus.cc:3037] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:20.791718 7825 raft_consensus.cc:504] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } }
I20260926 20:16:20.792075 7825 leader_election.cc:288] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [CANDIDATE]: Term 1 election: Requested vote from peers 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457), 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937)
I20260926 20:16:20.792528 7462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ba4486e7e6450499b95874f71290d1" candidate_uuid: "555a74f1a076438db59c7dc87eb72db1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5af0faa4e4465d8b19b495fc7cd943"
I20260926 20:16:20.792692 7744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ba4486e7e6450499b95874f71290d1" candidate_uuid: "555a74f1a076438db59c7dc87eb72db1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78efdddf1aff4afe9d5c59f5c9e36672"
I20260926 20:16:20.792692 7462 raft_consensus.cc:3037] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:20.792847 7744 raft_consensus.cc:3037] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:20.793936 7462 raft_consensus.cc:2454] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 555a74f1a076438db59c7dc87eb72db1 in term 1.
I20260926 20:16:20.794353 7744 raft_consensus.cc:2454] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 555a74f1a076438db59c7dc87eb72db1 in term 1.
I20260926 20:16:20.794430 7537 leader_election.cc:302] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5af0faa4e4465d8b19b495fc7cd943, 555a74f1a076438db59c7dc87eb72db1; no voters:
I20260926 20:16:20.794651 7825 raft_consensus.cc:2781] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:20.794888 7825 raft_consensus.cc:686] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 1 LEADER]: Becoming Leader. State: Replica: 555a74f1a076438db59c7dc87eb72db1, State: Running, Role: LEADER
I20260926 20:16:20.795317 7825 consensus_queue.cc:237] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } }
I20260926 20:16:20.798892 7302 catalog_manager.cc:5495] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194). New cstate: current_term: 1 leader_uuid: "555a74f1a076438db59c7dc87eb72db1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } health_report { overall_health: UNKNOWN } } }
I20260926 20:16:20.826330 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.822438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.31.119.194:46023)
0926 20:16:20.822574 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.822577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.822593 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:16:20.822686 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.822689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.822809 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.822906 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:20.823213 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.823929 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.824755 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.825008 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.825095 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.825482 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:20.825962 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:20.826068 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.826162 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.826215 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:16:20.832660 7744 raft_consensus.cc:1261] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Refusing update from remote peer 555a74f1a076438db59c7dc87eb72db1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:20.832660 7462 raft_consensus.cc:1261] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Refusing update from remote peer 555a74f1a076438db59c7dc87eb72db1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:20.833451 7829 consensus_queue.cc:1035] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:20.833698 7825 consensus_queue.cc:1035] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:20.851001 7302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:20.851209 7302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:20.853946 7302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f: Sending DeleteTablet for 3 replicas of tablet 32ba4486e7e6450499b95874f71290d1
I20260926 20:16:20.854833 7724 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ba4486e7e6450499b95874f71290d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:20 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:42004
I20260926 20:16:20.854943 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ba4486e7e6450499b95874f71290d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:20 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:51410
I20260926 20:16:20.855834 7842 tablet_replica.cc:330] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: stopping tablet replica
I20260926 20:16:20.856061 7302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:16:20.856238 7842 raft_consensus.cc:2229] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:16:20.856249 7442 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ba4486e7e6450499b95874f71290d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:20 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:48490
W20260926 20:16:20.856494 7302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:16:20.856707 7842 raft_consensus.cc:2258] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:20.855786 7843 tablet_replica.cc:330] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: stopping tablet replica
I20260926 20:16:20.857188 7843 raft_consensus.cc:2229] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:20.857378 7844 tablet_replica.cc:330] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: stopping tablet replica
I20260926 20:16:20.857575 7843 raft_consensus.cc:2258] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:20.857683 7844 raft_consensus.cc:2229] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:20.857968 7844 raft_consensus.cc:2258] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:20.859607 7844 ts_tablet_manager.cc:1905] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:20.862298 7442 tablet_service.cc:1467] Processing CreateTablet for tablet f318c3f89caa4495905172139b6cfa38 (DEFAULT_TABLE table=test-table [id=5f303704d60c45f28223fd6e85fc98cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:20.862298 7724 tablet_service.cc:1467] Processing CreateTablet for tablet f318c3f89caa4495905172139b6cfa38 (DEFAULT_TABLE table=test-table [id=5f303704d60c45f28223fd6e85fc98cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:20.862573 7442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f318c3f89caa4495905172139b6cfa38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:20.862573 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f318c3f89caa4495905172139b6cfa38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:20.862833 7843 ts_tablet_manager.cc:1905] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:20.865114 7819 tablet_bootstrap.cc:492] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: Bootstrap starting.
I20260926 20:16:20.865303 7583 tablet_service.cc:1467] Processing CreateTablet for tablet f318c3f89caa4495905172139b6cfa38 (DEFAULT_TABLE table=test-table [id=5f303704d60c45f28223fd6e85fc98cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:20.865682 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f318c3f89caa4495905172139b6cfa38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:20.866492 7819 tablet_bootstrap.cc:654] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:20.867745 7819 tablet_bootstrap.cc:492] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: No bootstrap required, opened a new log
I20260926 20:16:20.867826 7819 ts_tablet_manager.cc:1397] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:20.868335 7820 tablet_bootstrap.cc:492] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: Bootstrap starting.
I20260926 20:16:20.868327 7819 raft_consensus.cc:348] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:20.868438 7819 raft_consensus.cc:374] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:20.868468 7819 raft_consensus.cc:729] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5af0faa4e4465d8b19b495fc7cd943, State: Initialized, Role: FOLLOWER
I20260926 20:16:20.868598 7819 consensus_queue.cc:260] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:20.868867 7819 ts_tablet_manager.cc:1428] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:20.869410 7820 tablet_bootstrap.cc:654] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:20.870460 7821 tablet_bootstrap.cc:492] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: Bootstrap starting.
I20260926 20:16:20.871389 7821 tablet_bootstrap.cc:654] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:20.871646 7844 ts_tablet_manager.cc:1918] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:20.871798 7844 log.cc:1196] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/32ba4486e7e6450499b95874f71290d1
I20260926 20:16:20.872803 7844 ts_tablet_manager.cc:1939] T 32ba4486e7e6450499b95874f71290d1 P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting consensus metadata
I20260926 20:16:20.873045 7821 tablet_bootstrap.cc:492] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: No bootstrap required, opened a new log
I20260926 20:16:20.873152 7821 ts_tablet_manager.cc:1397] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 20:16:20.873636 7821 raft_consensus.cc:348] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:20.873754 7821 raft_consensus.cc:374] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:20.873795 7821 raft_consensus.cc:729] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555a74f1a076438db59c7dc87eb72db1, State: Initialized, Role: FOLLOWER
I20260926 20:16:20.873934 7821 consensus_queue.cc:260] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:20.874214 7821 ts_tablet_manager.cc:1428] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:20.874821 7290 catalog_manager.cc:4841] TS 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937): tablet 32ba4486e7e6450499b95874f71290d1 (table test-table [id=b3c0e15bc2254ad29bd3d626f68a3c88]) successfully deleted
I20260926 20:16:20.875612 7842 ts_tablet_manager.cc:1905] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:20.876255 7820 tablet_bootstrap.cc:492] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: No bootstrap required, opened a new log
I20260926 20:16:20.876348 7820 ts_tablet_manager.cc:1397] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260926 20:16:20.876814 7820 raft_consensus.cc:348] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:20.876925 7820 raft_consensus.cc:374] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:20.876966 7820 raft_consensus.cc:729] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78efdddf1aff4afe9d5c59f5c9e36672, State: Initialized, Role: FOLLOWER
I20260926 20:16:20.877086 7820 consensus_queue.cc:260] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:20.877358 7820 ts_tablet_manager.cc:1428] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:16:20.879096 7843 ts_tablet_manager.cc:1918] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:20.879211 7843 log.cc:1196] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/32ba4486e7e6450499b95874f71290d1
I20260926 20:16:20.879536 7843 ts_tablet_manager.cc:1939] T 32ba4486e7e6450499b95874f71290d1 P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting consensus metadata
I20260926 20:16:20.880793 7289 catalog_manager.cc:4841] TS 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457): tablet 32ba4486e7e6450499b95874f71290d1 (table test-table [id=b3c0e15bc2254ad29bd3d626f68a3c88]) successfully deleted
I20260926 20:16:20.882172 7842 ts_tablet_manager.cc:1918] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:20.882294 7842 log.cc:1196] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/32ba4486e7e6450499b95874f71290d1
I20260926 20:16:20.882575 7842 ts_tablet_manager.cc:1939] T 32ba4486e7e6450499b95874f71290d1 P 555a74f1a076438db59c7dc87eb72db1: Deleting consensus metadata
I20260926 20:16:20.883711 7290 catalog_manager.cc:4841] TS 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023): tablet 32ba4486e7e6450499b95874f71290d1 (table test-table [id=b3c0e15bc2254ad29bd3d626f68a3c88]) successfully deleted
W20260926 20:16:20.907284 7791 tablet.cc:2369] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:16:20.918937 7650 tablet.cc:2369] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:16:20.925663 7826 raft_consensus.cc:482] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:16:20.925836 7826 raft_consensus.cc:504] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:20.927129 7826 leader_election.cc:288] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023), 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457)
I20260926 20:16:20.932476 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.927422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:46023 (local address 127.31.119.193:57557)
0926 20:16:20.927807 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.927822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.927998 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.928453 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.928455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.928467 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:20.928652 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.928657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.929810 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.929814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.931089 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.931108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.932004 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.932008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.932115 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.932141 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.932225 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.932298 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":79,"thread_start_us":96,"threads_started":1}
I20260926 20:16:20.932574 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.927784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:57557 (local address 127.31.119.194:46023)
0926 20:16:20.927905 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.927909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.927924 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:20.928152 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.928178 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.928239 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.928286 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:20.928775 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.929678 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.931231 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.931849 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.931981 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.932060 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.932254 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.932327 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:16:20.932660 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.927618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:33697 (local address 127.31.119.195:37457)
0926 20:16:20.927748 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.927753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.928125 (+ 372us) server_negotiation.cc:407] Connection header received
0926 20:16:20.928240 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.928244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.928299 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.928409 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:20.929215 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.929828 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.931351 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.931957 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.932060 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.932116 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.932442 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.932535 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:16:20.933247 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.927492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:37457 (local address 127.31.119.193:33697)
0926 20:16:20.928039 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.928053 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.928138 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.928428 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.928432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.928457 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:20.928650 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.928657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.929962 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.929966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.931170 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.931182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.932169 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.932171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.932300 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.932323 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.932855 (+ 532us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.933097 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":26,"spinlock_wait_cycles":16256,"thread_start_us":91,"threads_started":1}
I20260926 20:16:20.933495 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f318c3f89caa4495905172139b6cfa38" candidate_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555a74f1a076438db59c7dc87eb72db1" is_pre_election: true
I20260926 20:16:20.933701 7744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f318c3f89caa4495905172139b6cfa38" candidate_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" is_pre_election: true
I20260926 20:16:20.933753 7603 raft_consensus.cc:2454] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5af0faa4e4465d8b19b495fc7cd943 in term 0.
I20260926 20:16:20.933837 7744 raft_consensus.cc:2454] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5af0faa4e4465d8b19b495fc7cd943 in term 0.
I20260926 20:16:20.934213 7394 leader_election.cc:302] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e5af0faa4e4465d8b19b495fc7cd943, 555a74f1a076438db59c7dc87eb72db1, 78efdddf1aff4afe9d5c59f5c9e36672; no voters:
I20260926 20:16:20.934521 7826 raft_consensus.cc:2781] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:16:20.934582 7826 raft_consensus.cc:482] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:16:20.934621 7826 raft_consensus.cc:3037] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:20.935582 7826 raft_consensus.cc:504] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
W20260926 20:16:20.935820 7510 tablet.cc:2369] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:16:20.936024 7826 leader_election.cc:288] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [CANDIDATE]: Term 1 election: Requested vote from peers 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023), 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457)
I20260926 20:16:20.936451 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f318c3f89caa4495905172139b6cfa38" candidate_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555a74f1a076438db59c7dc87eb72db1"
I20260926 20:16:20.936597 7603 raft_consensus.cc:3037] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:20.936750 7744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f318c3f89caa4495905172139b6cfa38" candidate_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78efdddf1aff4afe9d5c59f5c9e36672"
I20260926 20:16:20.936861 7744 raft_consensus.cc:3037] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:20.937803 7603 raft_consensus.cc:2454] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5af0faa4e4465d8b19b495fc7cd943 in term 1.
I20260926 20:16:20.938076 7744 raft_consensus.cc:2454] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5af0faa4e4465d8b19b495fc7cd943 in term 1.
I20260926 20:16:20.938130 7395 leader_election.cc:302] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5af0faa4e4465d8b19b495fc7cd943, 555a74f1a076438db59c7dc87eb72db1; no voters:
I20260926 20:16:20.938299 7826 raft_consensus.cc:2781] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:20.938532 7826 raft_consensus.cc:686] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5af0faa4e4465d8b19b495fc7cd943, State: Running, Role: LEADER
I20260926 20:16:20.938823 7826 consensus_queue.cc:237] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:20.940949 7299 catalog_manager.cc:5495] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193). New cstate: current_term: 1 leader_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:16:20 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453779, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.254@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:20.990900 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.975223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.31.119.254:36527)
0926 20:16:20.975387 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:20.975392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:20.975526 (+ 134us) server_negotiation.cc:407] Connection header received
0926 20:16:20.975646 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:20.975650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:20.975703 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:20.975812 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:20.976516 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.977362 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.978219 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.978560 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.978692 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.986901 (+ 8209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:20.986905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:20.986907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:20.986931 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:20.988466 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.989170 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.989177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.989182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.989271 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:20.989693 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:20.989696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:20.989697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:20.989994 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:20.990106 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:20.990581 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 20:16:20.990775 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":64}
I20260926 20:16:20.990904 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:20.975120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:36527 (local address 127.0.0.1:56574)
0926 20:16:20.975415 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:20.975432 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:20.975550 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:20.975872 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:20.975874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:20.976277 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:20.976373 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.976379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.977485 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.977488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.978090 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:20.978098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:20.978701 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:20.978703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:20.978767 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:20.984920 (+ 6153us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:20.984936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:20.986722 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:20.988612 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.988625 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.988632 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.989013 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.989412 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:20.989416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:20.989418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:20.989536 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:20.990167 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:20.990176 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:20.990368 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 20:16:20.990499 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 20:16:20.990703 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":110,"threads_started":1}
I20260926 20:16:20.995671 7299 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:56574:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:16:20.996562 7299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:16:21.002527 7442 tablet_service.cc:1467] Processing CreateTablet for tablet 59ef80e1fd8249bcac65d2c7e4a76c9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53b6905e5a5f40ddab5b8c9929d7f51c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:16:21.002645 7583 tablet_service.cc:1467] Processing CreateTablet for tablet 59ef80e1fd8249bcac65d2c7e4a76c9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53b6905e5a5f40ddab5b8c9929d7f51c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:16:21.003019 7442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ef80e1fd8249bcac65d2c7e4a76c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:21.003086 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ef80e1fd8249bcac65d2c7e4a76c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:21.005867 7819 tablet_bootstrap.cc:492] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: Bootstrap starting.
I20260926 20:16:21.005905 7821 tablet_bootstrap.cc:492] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: Bootstrap starting.
I20260926 20:16:21.006376 7724 tablet_service.cc:1467] Processing CreateTablet for tablet 59ef80e1fd8249bcac65d2c7e4a76c9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53b6905e5a5f40ddab5b8c9929d7f51c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:16:21.006755 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ef80e1fd8249bcac65d2c7e4a76c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:21.006896 7819 tablet_bootstrap.cc:654] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:21.006899 7821 tablet_bootstrap.cc:654] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:21.008351 7819 tablet_bootstrap.cc:492] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: No bootstrap required, opened a new log
I20260926 20:16:21.008458 7819 ts_tablet_manager.cc:1397] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:21.008916 7819 raft_consensus.cc:348] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.009060 7819 raft_consensus.cc:374] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:21.009101 7819 raft_consensus.cc:729] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5af0faa4e4465d8b19b495fc7cd943, State: Initialized, Role: FOLLOWER
I20260926 20:16:21.009277 7819 consensus_queue.cc:260] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.009611 7826 tablet_replica.cc:439] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } }
I20260926 20:16:21.009745 7826 tablet_replica.cc:442] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:16:21.010067 7820 tablet_bootstrap.cc:492] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: Bootstrap starting.
I20260926 20:16:21.011068 7820 tablet_bootstrap.cc:654] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:21.011664 7821 tablet_bootstrap.cc:492] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: No bootstrap required, opened a new log
I20260926 20:16:21.011768 7821 ts_tablet_manager.cc:1397] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 20:16:21.011825 7819 ts_tablet_manager.cc:1428] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:21.012217 7821 raft_consensus.cc:348] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.012351 7821 raft_consensus.cc:374] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:21.012400 7821 raft_consensus.cc:729] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555a74f1a076438db59c7dc87eb72db1, State: Initialized, Role: FOLLOWER
I20260926 20:16:21.012544 7821 consensus_queue.cc:260] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.012817 7821 ts_tablet_manager.cc:1428] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:21.013046 7837 tablet_replica.cc:439] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } }
I20260926 20:16:21.013192 7837 tablet_replica.cc:442] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:16:21.014396 7820 tablet_bootstrap.cc:492] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: No bootstrap required, opened a new log
I20260926 20:16:21.014503 7820 ts_tablet_manager.cc:1397] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:16:21.014973 7820 raft_consensus.cc:348] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.015098 7820 raft_consensus.cc:374] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:21.015129 7820 raft_consensus.cc:729] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78efdddf1aff4afe9d5c59f5c9e36672, State: Initialized, Role: FOLLOWER
I20260926 20:16:21.015265 7820 consensus_queue.cc:260] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.015623 7820 ts_tablet_manager.cc:1428] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:16:21.015632 7827 tablet_replica.cc:439] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } }
I20260926 20:16:21.016203 7827 tablet_replica.cc:442] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:16:21.034030 7827 raft_consensus.cc:482] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:16:21.034192 7827 raft_consensus.cc:504] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.035305 7827 leader_election.cc:288] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937), 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023)
I20260926 20:16:21.039932 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.035741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:36937 (local address 127.31.119.195:33877)
0926 20:16:21.035898 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:21.035915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:21.036051 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:21.036357 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:21.036360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:21.036379 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:21.036494 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:21.036501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.037662 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.037665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:21.038524 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:21.038533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.039416 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.039418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:21.039580 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.039600 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:16:21.039711 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:16:21.039789 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":3}
I20260926 20:16:21.039932 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.035846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:33877 (local address 127.31.119.193:36937)
0926 20:16:21.036078 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:21.036083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:21.036101 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:16:21.036165 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:21.036168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:21.036219 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:21.036340 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:21.036620 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.037543 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.038651 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.039284 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.039415 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.039467 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:21.039732 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 20:16:21.039809 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 20:16:21.040465 7462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ef80e1fd8249bcac65d2c7e4a76c9f" candidate_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" is_pre_election: true
I20260926 20:16:21.040658 7462 raft_consensus.cc:2454] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78efdddf1aff4afe9d5c59f5c9e36672 in term 0.
I20260926 20:16:21.040822 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.036080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:46023 (local address 127.31.119.195:60597)
0926 20:16:21.036549 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:21.036560 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:21.036675 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:21.036940 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:21.036943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:21.036963 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:21.037543 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:21.037553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.038331 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.038337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:21.039562 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:21.039570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.040376 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.040379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:21.040487 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.040514 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:16:21.040606 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:16:21.040678 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":23,"thread_start_us":186,"threads_started":1}
I20260926 20:16:21.040868 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.036202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:60597 (local address 127.31.119.194:46023)
0926 20:16:21.036359 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:21.036364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:21.036638 (+ 274us) server_negotiation.cc:407] Connection header received
0926 20:16:21.036782 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:21.036785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:21.036834 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:21.036909 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:21.037708 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.038204 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.039697 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.040226 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.040390 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.040440 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:21.040657 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 20:16:21.040736 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:16:21.041091 7678 leader_election.cc:302] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5af0faa4e4465d8b19b495fc7cd943, 78efdddf1aff4afe9d5c59f5c9e36672; no voters:
I20260926 20:16:21.041492 7827 raft_consensus.cc:2781] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:16:21.041581 7827 raft_consensus.cc:482] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:16:21.041626 7827 raft_consensus.cc:3037] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:21.041823 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ef80e1fd8249bcac65d2c7e4a76c9f" candidate_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555a74f1a076438db59c7dc87eb72db1" is_pre_election: true
I20260926 20:16:21.042006 7603 raft_consensus.cc:2454] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78efdddf1aff4afe9d5c59f5c9e36672 in term 0.
I20260926 20:16:21.042892 7827 raft_consensus.cc:504] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.043260 7827 leader_election.cc:288] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [CANDIDATE]: Term 1 election: Requested vote from peers 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937), 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023)
I20260926 20:16:21.043717 7462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ef80e1fd8249bcac65d2c7e4a76c9f" candidate_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5af0faa4e4465d8b19b495fc7cd943"
I20260926 20:16:21.043836 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ef80e1fd8249bcac65d2c7e4a76c9f" candidate_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555a74f1a076438db59c7dc87eb72db1"
I20260926 20:16:21.043884 7462 raft_consensus.cc:3037] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:21.043968 7603 raft_consensus.cc:3037] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:21.045054 7603 raft_consensus.cc:2454] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78efdddf1aff4afe9d5c59f5c9e36672 in term 1.
I20260926 20:16:21.045068 7462 raft_consensus.cc:2454] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78efdddf1aff4afe9d5c59f5c9e36672 in term 1.
I20260926 20:16:21.045392 7678 leader_election.cc:302] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555a74f1a076438db59c7dc87eb72db1, 78efdddf1aff4afe9d5c59f5c9e36672; no voters:
I20260926 20:16:21.045733 7827 raft_consensus.cc:2781] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:21.045970 7827 raft_consensus.cc:686] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 LEADER]: Becoming Leader. State: Replica: 78efdddf1aff4afe9d5c59f5c9e36672, State: Running, Role: LEADER
I20260926 20:16:21.046291 7827 consensus_queue.cc:237] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.048187 7862 tablet_replica.cc:439] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: TxnStatusTablet state changed. Reason: New leader 78efdddf1aff4afe9d5c59f5c9e36672. Latest consensus state: current_term: 1 leader_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } }
I20260926 20:16:21.048357 7862 tablet_replica.cc:442] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:16:21.049214 7864 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:16:21.049168 7302 catalog_manager.cc:5495] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195). New cstate: current_term: 1 leader_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } health_report { overall_health: HEALTHY } } }
I20260926 20:16:21.049391 7864 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:16:21.069749 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.064796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.31.119.195:37457)
0926 20:16:21.064917 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:21.064922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:21.064937 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:21.065112 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:21.065119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:21.065191 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:21.065657 (+ 466us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:21.065758 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.066552 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.067797 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.068189 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.068279 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.068648 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:21.069276 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:21.069430 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:21.069530 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 20:16:21.069591 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:16:21.069777 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.064687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:37457 (local address 127.0.0.1:42008)
0926 20:16:21.064829 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:21.064846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:21.064961 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:21.065388 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:21.065392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:21.065400 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:16:21.065532 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:21.065539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.066692 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.066695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:21.067680 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:21.067691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.068304 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.068307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:21.068407 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.068469 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:16:21.069418 (+ 949us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:16:21.069440 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:16:21.069521 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:16:21.069634 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:16:21.075410 7462 raft_consensus.cc:1261] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Refusing update from remote peer 78efdddf1aff4afe9d5c59f5c9e36672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:21.075572 7603 raft_consensus.cc:1261] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Refusing update from remote peer 78efdddf1aff4afe9d5c59f5c9e36672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:21.076030 7827 consensus_queue.cc:1035] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:21.076460 7862 consensus_queue.cc:1035] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:21.077803 7826 tablet_replica.cc:439] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: TxnStatusTablet state changed. Reason: New leader 78efdddf1aff4afe9d5c59f5c9e36672. Latest consensus state: current_term: 1 leader_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } }
I20260926 20:16:21.077929 7826 tablet_replica.cc:442] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:16:21.078153 7837 tablet_replica.cc:439] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: TxnStatusTablet state changed. Reason: New leader 78efdddf1aff4afe9d5c59f5c9e36672. Latest consensus state: current_term: 1 leader_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } }
I20260926 20:16:21.078290 7837 tablet_replica.cc:442] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:16:21.080320 7862 tablet_replica.cc:439] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } }
I20260926 20:16:21.080440 7862 tablet_replica.cc:442] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:16:21.081167 7826 tablet_replica.cc:439] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } }
I20260926 20:16:21.081307 7826 tablet_replica.cc:442] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 1e5af0faa4e4465d8b19b495fc7cd943: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:16:21.091080 7827 tablet_replica.cc:439] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } }
I20260926 20:16:21.091266 7827 tablet_replica.cc:442] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 78efdddf1aff4afe9d5c59f5c9e36672: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:16:21.092047 7837 tablet_replica.cc:439] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } } }
I20260926 20:16:21.092168 7837 tablet_replica.cc:442] T 59ef80e1fd8249bcac65d2c7e4a76c9f P 555a74f1a076438db59c7dc87eb72db1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:16:21.094751 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.086768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.31.119.193:36937)
0926 20:16:21.086910 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:21.086916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:21.086934 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:16:21.087135 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:21.087141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:21.087228 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:21.087353 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:21.087620 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.088549 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.089909 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.090740 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.092902 (+ 2162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.093007 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:21.093612 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:21.094489 (+ 877us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:21.094525 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 20:16:21.094591 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:16:21.108448 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.103132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:37457 (local address 127.31.119.193:52409)
0926 20:16:21.103493 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:21.103511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:21.103633 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:21.103908 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:21.103913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:21.103925 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:16:21.104099 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:21.104109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.105157 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.105161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:21.106786 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:21.106801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.107547 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.107553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:21.107653 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.107716 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:16:21.108088 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:16:21.108095 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:16:21.108163 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:16:21.108253 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":115,"threads_started":1}
I20260926 20:16:21.108459 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.103521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:52409 (local address 127.31.119.195:37457)
0926 20:16:21.103637 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:21.103642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:21.103657 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:21.103740 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:21.103744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:21.103797 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:21.103877 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:21.104230 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.105021 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.106922 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.107370 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.107511 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.107877 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:21.107986 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:21.108070 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:21.108181 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 20:16:21.108254 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:16:21.116097 7744 raft_consensus.cc:1261] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Refusing update from remote peer 1e5af0faa4e4465d8b19b495fc7cd943: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:21.116196 7603 raft_consensus.cc:1261] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Refusing update from remote peer 1e5af0faa4e4465d8b19b495fc7cd943: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:21.116626 7826 consensus_queue.cc:1035] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:21.117058 7826 consensus_queue.cc:1035] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:21.120611 7832 mvcc.cc:205] Tried to move back new op lower bound from 7333698687447076864 to 7333698686727897088. Current Snapshot: MvccSnapshot[applied={T|T < 7333698687447076864}]
I20260926 20:16:21.124783 7834 mvcc.cc:205] Tried to move back new op lower bound from 7333698687447076864 to 7333698686727897088. Current Snapshot: MvccSnapshot[applied={T|T < 7333698687447076864}]
I20260926 20:16:21.141726 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.137489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:39277 (local address 127.31.119.193:36937)
0926 20:16:21.137933 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:21.137937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:21.137950 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:16:21.138006 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:21.138009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:21.138060 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:21.138126 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:21.138497 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.139142 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.140370 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.140668 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.140767 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.141202 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:21.141352 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:21.141430 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:21.141563 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 20:16:21.141613 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
I20260926 20:16:21.141799 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.137410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:36937 (local address 127.31.119.195:39277)
0926 20:16:21.137672 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:21.137688 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:21.137775 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:21.138239 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:21.138242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:21.138248 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:16:21.138377 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:21.138383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.139251 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.139255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:21.140249 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:21.140259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.140855 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.140858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:21.140961 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.141022 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:16:21.141447 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:16:21.141455 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:16:21.141535 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:16:21.141634 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 20:16:21.172044 7299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:21.172242 7299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:21.174834 7299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f: Sending DeleteTablet for 3 replicas of tablet f318c3f89caa4495905172139b6cfa38
I20260926 20:16:21.175683 7442 tablet_service.cc:1514] Processing DeleteTablet for tablet f318c3f89caa4495905172139b6cfa38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:48490
I20260926 20:16:21.175895 7844 tablet_replica.cc:330] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: stopping tablet replica
I20260926 20:16:21.175961 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet f318c3f89caa4495905172139b6cfa38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:51410
I20260926 20:16:21.176026 7844 raft_consensus.cc:2229] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:16:21.176162 7842 tablet_replica.cc:330] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: stopping tablet replica
I20260926 20:16:21.176301 7842 raft_consensus.cc:2229] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:21.176333 7844 raft_consensus.cc:2258] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:21.176429 7842 raft_consensus.cc:2258] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:21.177309 7842 ts_tablet_manager.cc:1905] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:21.177397 7844 ts_tablet_manager.cc:1905] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:21.180302 7842 ts_tablet_manager.cc:1918] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:16:21.180362 7844 ts_tablet_manager.cc:1918] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:16:21.180395 7842 log.cc:1196] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/f318c3f89caa4495905172139b6cfa38
I20260926 20:16:21.180441 7844 log.cc:1196] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/f318c3f89caa4495905172139b6cfa38
I20260926 20:16:21.180666 7842 ts_tablet_manager.cc:1939] T f318c3f89caa4495905172139b6cfa38 P 555a74f1a076438db59c7dc87eb72db1: Deleting consensus metadata
I20260926 20:16:21.180716 7844 ts_tablet_manager.cc:1939] T f318c3f89caa4495905172139b6cfa38 P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting consensus metadata
I20260926 20:16:21.181550 7290 catalog_manager.cc:4841] TS 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937): tablet f318c3f89caa4495905172139b6cfa38 (table test-table [id=5f303704d60c45f28223fd6e85fc98cf]) successfully deleted
I20260926 20:16:21.181769 7290 catalog_manager.cc:4841] TS 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023): tablet f318c3f89caa4495905172139b6cfa38 (table test-table [id=5f303704d60c45f28223fd6e85fc98cf]) successfully deleted
I20260926 20:16:21.182127 7724 tablet_service.cc:1514] Processing DeleteTablet for tablet f318c3f89caa4495905172139b6cfa38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:42004
I20260926 20:16:21.182358 7843 tablet_replica.cc:330] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: stopping tablet replica
I20260926 20:16:21.182479 7843 raft_consensus.cc:2229] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:21.182602 7843 raft_consensus.cc:2258] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:21.183758 7843 ts_tablet_manager.cc:1905] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:21.186748 7843 ts_tablet_manager.cc:1918] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:16:21.186823 7843 log.cc:1196] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/f318c3f89caa4495905172139b6cfa38
I20260926 20:16:21.187074 7843 ts_tablet_manager.cc:1939] T f318c3f89caa4495905172139b6cfa38 P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting consensus metadata
I20260926 20:16:21.188186 7289 catalog_manager.cc:4841] TS 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457): tablet f318c3f89caa4495905172139b6cfa38 (table test-table [id=5f303704d60c45f28223fd6e85fc98cf]) successfully deleted
Sep 26 20:16:21 dist-test-slave-vnbh krb5kdc[7258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:16:21 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:21 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:16:21.227236 7352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:21.218752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.31.119.254:36527)
0926 20:16:21.218881 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:21.218888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:21.219109 (+ 221us) server_negotiation.cc:407] Connection header received
0926 20:16:21.219278 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:21.219282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:21.219343 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:21.219441 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:21.220443 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.221315 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.222759 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.223186 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.223313 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.227095 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56580: BlockingRecv error: recv got EOF from 127.0.0.1:56580 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 20:16:21 dist-test-slave-vnbh krb5kdc[7258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM
I20260926 20:16:21.240509 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.229455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.31.119.254:36527)
0926 20:16:21.229581 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:21.229586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:21.230005 (+ 419us) server_negotiation.cc:407] Connection header received
0926 20:16:21.230120 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:21.230123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:21.230174 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:21.230260 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:21.230978 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.231779 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.233119 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.233511 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.233620 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.236568 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:21.236573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:21.236576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:21.236606 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:21.238462 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:21.238976 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:21.238980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:21.238982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:21.239103 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:21.239497 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:21.239501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:21.239503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:21.239754 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:21.239892 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:21.240189 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 20:16:21.240322 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:16:21.243525 7299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:16:21.244028 7299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:16:21.249471 7442 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdfd218cb194d09946c1cded4f8ae61 (DEFAULT_TABLE table=test-table [id=63b0389f21084ee5b5104dfcf2b958bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:21.249631 7583 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdfd218cb194d09946c1cded4f8ae61 (DEFAULT_TABLE table=test-table [id=63b0389f21084ee5b5104dfcf2b958bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:21.249816 7442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdfd218cb194d09946c1cded4f8ae61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:21.249938 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdfd218cb194d09946c1cded4f8ae61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:21.251775 7819 tablet_bootstrap.cc:492] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943: Bootstrap starting.
I20260926 20:16:21.252030 7724 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdfd218cb194d09946c1cded4f8ae61 (DEFAULT_TABLE table=test-table [id=63b0389f21084ee5b5104dfcf2b958bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:21.252231 7821 tablet_bootstrap.cc:492] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1: Bootstrap starting.
I20260926 20:16:21.252293 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdfd218cb194d09946c1cded4f8ae61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:21.252763 7819 tablet_bootstrap.cc:654] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:21.253062 7821 tablet_bootstrap.cc:654] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:21.254195 7821 tablet_bootstrap.cc:492] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1: No bootstrap required, opened a new log
I20260926 20:16:21.254312 7821 ts_tablet_manager.cc:1397] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:16:21.254804 7820 tablet_bootstrap.cc:492] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672: Bootstrap starting.
I20260926 20:16:21.254756 7821 raft_consensus.cc:348] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.254877 7821 raft_consensus.cc:374] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:21.254920 7821 raft_consensus.cc:729] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555a74f1a076438db59c7dc87eb72db1, State: Initialized, Role: FOLLOWER
I20260926 20:16:21.255051 7821 consensus_queue.cc:260] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.255390 7821 ts_tablet_manager.cc:1428] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:16:21.255617 7820 tablet_bootstrap.cc:654] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:21.255865 7819 tablet_bootstrap.cc:492] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943: No bootstrap required, opened a new log
I20260926 20:16:21.255927 7819 ts_tablet_manager.cc:1397] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:16:21.256413 7819 raft_consensus.cc:348] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.256644 7819 raft_consensus.cc:374] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:21.256686 7819 raft_consensus.cc:729] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5af0faa4e4465d8b19b495fc7cd943, State: Initialized, Role: FOLLOWER
I20260926 20:16:21.256702 7820 tablet_bootstrap.cc:492] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672: No bootstrap required, opened a new log
I20260926 20:16:21.256778 7820 ts_tablet_manager.cc:1397] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 20:16:21.256816 7819 consensus_queue.cc:260] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.257067 7819 ts_tablet_manager.cc:1428] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:21.257171 7820 raft_consensus.cc:348] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.257324 7820 raft_consensus.cc:374] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:21.257357 7820 raft_consensus.cc:729] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78efdddf1aff4afe9d5c59f5c9e36672, State: Initialized, Role: FOLLOWER
I20260926 20:16:21.257481 7820 consensus_queue.cc:260] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.257730 7820 ts_tablet_manager.cc:1428] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:16:21.354566 7874 raft_consensus.cc:482] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:16:21.354715 7874 raft_consensus.cc:504] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.355122 7874 leader_election.cc:288] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937), 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023)
I20260926 20:16:21.355664 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdfd218cb194d09946c1cded4f8ae61" candidate_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555a74f1a076438db59c7dc87eb72db1" is_pre_election: true
I20260926 20:16:21.355664 7462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdfd218cb194d09946c1cded4f8ae61" candidate_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" is_pre_election: true
I20260926 20:16:21.355832 7462 raft_consensus.cc:2454] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78efdddf1aff4afe9d5c59f5c9e36672 in term 0.
I20260926 20:16:21.355832 7603 raft_consensus.cc:2454] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78efdddf1aff4afe9d5c59f5c9e36672 in term 0.
I20260926 20:16:21.356202 7678 leader_election.cc:302] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555a74f1a076438db59c7dc87eb72db1, 78efdddf1aff4afe9d5c59f5c9e36672; no voters:
I20260926 20:16:21.356524 7874 raft_consensus.cc:2781] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:16:21.356580 7874 raft_consensus.cc:482] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:16:21.356630 7874 raft_consensus.cc:3037] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:21.357973 7874 raft_consensus.cc:504] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.358393 7874 leader_election.cc:288] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [CANDIDATE]: Term 1 election: Requested vote from peers 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937), 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023)
I20260926 20:16:21.358781 7462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdfd218cb194d09946c1cded4f8ae61" candidate_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5af0faa4e4465d8b19b495fc7cd943"
I20260926 20:16:21.358927 7462 raft_consensus.cc:3037] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:21.359855 7462 raft_consensus.cc:2454] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78efdddf1aff4afe9d5c59f5c9e36672 in term 1.
I20260926 20:16:21.360162 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdfd218cb194d09946c1cded4f8ae61" candidate_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555a74f1a076438db59c7dc87eb72db1"
I20260926 20:16:21.360210 7678 leader_election.cc:302] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5af0faa4e4465d8b19b495fc7cd943, 78efdddf1aff4afe9d5c59f5c9e36672; no voters:
I20260926 20:16:21.360324 7603 raft_consensus.cc:3037] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:21.360381 7874 raft_consensus.cc:2781] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:21.360459 7874 raft_consensus.cc:686] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 LEADER]: Becoming Leader. State: Replica: 78efdddf1aff4afe9d5c59f5c9e36672, State: Running, Role: LEADER
I20260926 20:16:21.360587 7874 consensus_queue.cc:237] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.361661 7603 raft_consensus.cc:2454] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78efdddf1aff4afe9d5c59f5c9e36672 in term 1.
I20260926 20:16:21.363128 7302 catalog_manager.cc:5495] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195). New cstate: current_term: 1 leader_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } health_report { overall_health: HEALTHY } } }
I20260926 20:16:21.386162 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.382586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.31.119.195:37457)
0926 20:16:21.382723 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:21.382726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:21.382738 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:16:21.382831 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:21.382833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:21.382887 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:21.382964 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:21.383496 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.384104 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.385092 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.385340 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.385446 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.385731 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:21.385817 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:21.385877 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:21.385980 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 20:16:21.386032 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:16:21.388837 7603 raft_consensus.cc:1261] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Refusing update from remote peer 78efdddf1aff4afe9d5c59f5c9e36672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:21.389046 7462 raft_consensus.cc:1261] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Refusing update from remote peer 78efdddf1aff4afe9d5c59f5c9e36672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:21.389400 7862 consensus_queue.cc:1035] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:21.389626 7874 consensus_queue.cc:1035] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:21.400341 7299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:21.400539 7299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:21.403328 7299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f: Sending DeleteTablet for 3 replicas of tablet 4cdfd218cb194d09946c1cded4f8ae61
I20260926 20:16:21.403914 7724 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdfd218cb194d09946c1cded4f8ae61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:42004
I20260926 20:16:21.404161 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdfd218cb194d09946c1cded4f8ae61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:51410
I20260926 20:16:21.404299 7843 tablet_replica.cc:330] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672: stopping tablet replica
I20260926 20:16:21.404441 7843 raft_consensus.cc:2229] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:16:21.404752 7843 raft_consensus.cc:2258] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:21.404963 7842 tablet_replica.cc:330] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1: stopping tablet replica
I20260926 20:16:21.405102 7842 raft_consensus.cc:2229] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:21.405263 7842 raft_consensus.cc:2258] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:21.406080 7843 ts_tablet_manager.cc:1905] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:21.406157 7724 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdfd218cb194d09946c1cded4f8ae61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:42004
I20260926 20:16:21.406158 7442 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdfd218cb194d09946c1cded4f8ae61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:48490
I20260926 20:16:21.406327 7844 tablet_replica.cc:330] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943: stopping tablet replica
I20260926 20:16:21.406440 7844 raft_consensus.cc:2229] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:21.406569 7844 raft_consensus.cc:2258] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:21.406805 7842 ts_tablet_manager.cc:1905] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:21.407155 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdfd218cb194d09946c1cded4f8ae61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:51410
I20260926 20:16:21.408552 7843 ts_tablet_manager.cc:1918] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:21.408916 7843 log.cc:1196] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/4cdfd218cb194d09946c1cded4f8ae61
I20260926 20:16:21.409041 7842 ts_tablet_manager.cc:1918] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:21.409119 7842 log.cc:1196] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/4cdfd218cb194d09946c1cded4f8ae61
I20260926 20:16:21.409194 7843 ts_tablet_manager.cc:1939] T 4cdfd218cb194d09946c1cded4f8ae61 P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting consensus metadata
I20260926 20:16:21.409507 7842 ts_tablet_manager.cc:1939] T 4cdfd218cb194d09946c1cded4f8ae61 P 555a74f1a076438db59c7dc87eb72db1: Deleting consensus metadata
I20260926 20:16:21.409932 7289 catalog_manager.cc:4841] TS 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457): tablet 4cdfd218cb194d09946c1cded4f8ae61 (table test-table [id=63b0389f21084ee5b5104dfcf2b958bd]) successfully deleted
I20260926 20:16:21.410236 7290 catalog_manager.cc:4841] TS 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023): tablet 4cdfd218cb194d09946c1cded4f8ae61 (table test-table [id=63b0389f21084ee5b5104dfcf2b958bd]) successfully deleted
W20260926 20:16:21.410578 7290 catalog_manager.cc:4808] TS 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023): delete failed for tablet 4cdfd218cb194d09946c1cded4f8ae61 because the tablet was not found. No further retry: Not found: Tablet not found: 4cdfd218cb194d09946c1cded4f8ae61
W20260926 20:16:21.410579 7289 catalog_manager.cc:4808] TS 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457): delete failed for tablet 4cdfd218cb194d09946c1cded4f8ae61 because the tablet was not found. No further retry: Not found: Tablet not found: 4cdfd218cb194d09946c1cded4f8ae61
I20260926 20:16:21.410945 7299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:16:21.411163 7844 ts_tablet_manager.cc:1905] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:16:21.411412 7299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:16:21.413600 7844 ts_tablet_manager.cc:1918] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:21.413682 7844 log.cc:1196] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/4cdfd218cb194d09946c1cded4f8ae61
I20260926 20:16:21.413924 7844 ts_tablet_manager.cc:1939] T 4cdfd218cb194d09946c1cded4f8ae61 P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting consensus metadata
I20260926 20:16:21.415082 7290 catalog_manager.cc:4841] TS 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937): tablet 4cdfd218cb194d09946c1cded4f8ae61 (table test-table [id=63b0389f21084ee5b5104dfcf2b958bd]) successfully deleted
I20260926 20:16:21.416532 7442 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5edd40c24c45a4a2e82c4678a1424f (DEFAULT_TABLE table=test-table [id=0a003084691a4282a5033412d900f62e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:21.416815 7442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5edd40c24c45a4a2e82c4678a1424f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:21.418637 7819 tablet_bootstrap.cc:492] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943: Bootstrap starting.
I20260926 20:16:21.419209 7724 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5edd40c24c45a4a2e82c4678a1424f (DEFAULT_TABLE table=test-table [id=0a003084691a4282a5033412d900f62e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:21.419472 7819 tablet_bootstrap.cc:654] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:21.419472 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5edd40c24c45a4a2e82c4678a1424f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:21.420837 7819 tablet_bootstrap.cc:492] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943: No bootstrap required, opened a new log
I20260926 20:16:21.420926 7819 ts_tablet_manager.cc:1397] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:16:21.421360 7819 raft_consensus.cc:348] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.421567 7819 raft_consensus.cc:374] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:21.421612 7819 raft_consensus.cc:729] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5af0faa4e4465d8b19b495fc7cd943, State: Initialized, Role: FOLLOWER
I20260926 20:16:21.421769 7819 consensus_queue.cc:260] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.421931 7820 tablet_bootstrap.cc:492] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672: Bootstrap starting.
I20260926 20:16:21.422046 7819 ts_tablet_manager.cc:1428] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:16:21.422690 7820 tablet_bootstrap.cc:654] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:21.423849 7583 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5edd40c24c45a4a2e82c4678a1424f (DEFAULT_TABLE table=test-table [id=0a003084691a4282a5033412d900f62e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:21.424139 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5edd40c24c45a4a2e82c4678a1424f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:21.426110 7821 tablet_bootstrap.cc:492] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1: Bootstrap starting.
I20260926 20:16:21.426905 7821 tablet_bootstrap.cc:654] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:21.428041 7821 tablet_bootstrap.cc:492] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1: No bootstrap required, opened a new log
I20260926 20:16:21.428128 7821 ts_tablet_manager.cc:1397] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:16:21.428532 7821 raft_consensus.cc:348] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.428691 7821 raft_consensus.cc:374] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:21.428727 7821 raft_consensus.cc:729] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555a74f1a076438db59c7dc87eb72db1, State: Initialized, Role: FOLLOWER
I20260926 20:16:21.428845 7821 consensus_queue.cc:260] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.429078 7821 ts_tablet_manager.cc:1428] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 20:16:21.430073 7820 tablet_bootstrap.cc:492] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672: No bootstrap required, opened a new log
I20260926 20:16:21.430166 7820 ts_tablet_manager.cc:1397] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260926 20:16:21.430548 7820 raft_consensus.cc:348] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.430672 7820 raft_consensus.cc:374] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:21.430708 7820 raft_consensus.cc:729] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78efdddf1aff4afe9d5c59f5c9e36672, State: Initialized, Role: FOLLOWER
I20260926 20:16:21.430825 7820 consensus_queue.cc:260] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.431093 7820 ts_tablet_manager.cc:1428] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:16:21.484316 7876 raft_consensus.cc:482] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:16:21.484481 7876 raft_consensus.cc:504] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.484961 7876 leader_election.cc:288] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023), 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457)
I20260926 20:16:21.485452 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5edd40c24c45a4a2e82c4678a1424f" candidate_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555a74f1a076438db59c7dc87eb72db1" is_pre_election: true
I20260926 20:16:21.485462 7744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5edd40c24c45a4a2e82c4678a1424f" candidate_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" is_pre_election: true
I20260926 20:16:21.485613 7603 raft_consensus.cc:2454] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5af0faa4e4465d8b19b495fc7cd943 in term 0.
I20260926 20:16:21.485683 7744 raft_consensus.cc:2454] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5af0faa4e4465d8b19b495fc7cd943 in term 0.
I20260926 20:16:21.485952 7395 leader_election.cc:302] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5af0faa4e4465d8b19b495fc7cd943, 555a74f1a076438db59c7dc87eb72db1; no voters:
I20260926 20:16:21.486166 7876 raft_consensus.cc:2781] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:16:21.486225 7876 raft_consensus.cc:482] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:16:21.486253 7876 raft_consensus.cc:3037] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:21.487572 7876 raft_consensus.cc:504] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.487937 7876 leader_election.cc:288] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [CANDIDATE]: Term 1 election: Requested vote from peers 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023), 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457)
I20260926 20:16:21.488282 7744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5edd40c24c45a4a2e82c4678a1424f" candidate_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78efdddf1aff4afe9d5c59f5c9e36672"
I20260926 20:16:21.488286 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5edd40c24c45a4a2e82c4678a1424f" candidate_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555a74f1a076438db59c7dc87eb72db1"
I20260926 20:16:21.488411 7603 raft_consensus.cc:3037] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:21.488428 7744 raft_consensus.cc:3037] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:21.489454 7744 raft_consensus.cc:2454] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5af0faa4e4465d8b19b495fc7cd943 in term 1.
I20260926 20:16:21.489617 7603 raft_consensus.cc:2454] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5af0faa4e4465d8b19b495fc7cd943 in term 1.
I20260926 20:16:21.489771 7394 leader_election.cc:302] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5af0faa4e4465d8b19b495fc7cd943, 78efdddf1aff4afe9d5c59f5c9e36672; no voters:
I20260926 20:16:21.489943 7876 raft_consensus.cc:2781] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:21.490051 7876 raft_consensus.cc:686] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5af0faa4e4465d8b19b495fc7cd943, State: Running, Role: LEADER
I20260926 20:16:21.490198 7876 consensus_queue.cc:237] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } }
I20260926 20:16:21.492129 7302 catalog_manager.cc:5495] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193). New cstate: current_term: 1 leader_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5af0faa4e4465d8b19b495fc7cd943" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 } health_report { overall_health: UNKNOWN } } }
I20260926 20:16:21.517743 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:21.514284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.31.119.193:36937)
0926 20:16:21.514409 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:21.514413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:21.514429 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:16:21.514572 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:21.514575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:21.514626 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:21.514715 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:21.514911 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.515607 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.516571 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:21.516805 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:21.516920 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:21.517183 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:21.517267 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:21.517401 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:21.517530 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 20:16:21.517593 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:16:21.524946 7744 raft_consensus.cc:1261] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Refusing update from remote peer 1e5af0faa4e4465d8b19b495fc7cd943: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:21.524959 7603 raft_consensus.cc:1261] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Refusing update from remote peer 1e5af0faa4e4465d8b19b495fc7cd943: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:21.525408 7826 consensus_queue.cc:1035] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555a74f1a076438db59c7dc87eb72db1" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:21.525604 7876 consensus_queue.cc:1035] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78efdddf1aff4afe9d5c59f5c9e36672" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:21.564514 7299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:21.564731 7299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:21.567242 7299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c3caba2ac7a4cfd94b3b07c0008d70f: Sending DeleteTablet for 3 replicas of tablet 9e5edd40c24c45a4a2e82c4678a1424f
I20260926 20:16:21.567842 7442 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5edd40c24c45a4a2e82c4678a1424f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:48490
I20260926 20:16:21.567906 7724 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5edd40c24c45a4a2e82c4678a1424f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:42004
I20260926 20:16:21.568089 7844 tablet_replica.cc:330] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943: stopping tablet replica
I20260926 20:16:21.568225 7844 raft_consensus.cc:2229] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:16:21.568425 7844 raft_consensus.cc:2258] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:21.568641 7843 tablet_replica.cc:330] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672: stopping tablet replica
I20260926 20:16:21.568758 7843 raft_consensus.cc:2229] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:21.568862 7843 raft_consensus.cc:2258] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:21.569042 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5edd40c24c45a4a2e82c4678a1424f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:21 UTC) from {username='oryx', principal='oryx/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:51410
I20260926 20:16:21.569211 7842 tablet_replica.cc:330] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1: stopping tablet replica
I20260926 20:16:21.569347 7842 raft_consensus.cc:2229] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:21.569456 7842 raft_consensus.cc:2258] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:21.569658 7843 ts_tablet_manager.cc:1905] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:21.570042 7844 ts_tablet_manager.cc:1905] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:21.570212 7842 ts_tablet_manager.cc:1905] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:21.572320 7843 ts_tablet_manager.cc:1918] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:16:21.572382 7844 ts_tablet_manager.cc:1918] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:16:21.572397 7843 log.cc:1196] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/9e5edd40c24c45a4a2e82c4678a1424f
I20260926 20:16:21.572444 7844 log.cc:1196] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/9e5edd40c24c45a4a2e82c4678a1424f
I20260926 20:16:21.572681 7843 ts_tablet_manager.cc:1939] T 9e5edd40c24c45a4a2e82c4678a1424f P 78efdddf1aff4afe9d5c59f5c9e36672: Deleting consensus metadata
I20260926 20:16:21.572690 7844 ts_tablet_manager.cc:1939] T 9e5edd40c24c45a4a2e82c4678a1424f P 1e5af0faa4e4465d8b19b495fc7cd943: Deleting consensus metadata
I20260926 20:16:21.572759 7842 ts_tablet_manager.cc:1918] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:16:21.572829 7842 log.cc:1196] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/9e5edd40c24c45a4a2e82c4678a1424f
I20260926 20:16:21.573074 7842 ts_tablet_manager.cc:1939] T 9e5edd40c24c45a4a2e82c4678a1424f P 555a74f1a076438db59c7dc87eb72db1: Deleting consensus metadata
I20260926 20:16:21.573501 7289 catalog_manager.cc:4841] TS 78efdddf1aff4afe9d5c59f5c9e36672 (127.31.119.195:37457): tablet 9e5edd40c24c45a4a2e82c4678a1424f (table test-table [id=0a003084691a4282a5033412d900f62e]) successfully deleted
I20260926 20:16:21.573618 7290 catalog_manager.cc:4841] TS 1e5af0faa4e4465d8b19b495fc7cd943 (127.31.119.193:36937): tablet 9e5edd40c24c45a4a2e82c4678a1424f (table test-table [id=0a003084691a4282a5033412d900f62e]) successfully deleted
I20260926 20:16:21.573783 7290 catalog_manager.cc:4841] TS 555a74f1a076438db59c7dc87eb72db1 (127.31.119.194:46023): tablet 9e5edd40c24c45a4a2e82c4678a1424f (table test-table [id=0a003084691a4282a5033412d900f62e]) successfully deleted
I20260926 20:16:21.574607 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 7376
I20260926 20:16:21.584098 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 7518
I20260926 20:16:21.591315 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 7658
I20260926 20:16:21.598469 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 7271
2026-09-26T20:16:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3119 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:16:21 dist-test-slave-vnbh krb5kdc[7907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:16:21 dist-test-slave-vnbh krb5kdc[7907](info): set up 2 sockets
Sep 26 20:16:21 dist-test-slave-vnbh krb5kdc[7907](info): commencing operation
krb5kdc: starting...
W20260926 20:16:22.658980 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:22 dist-test-slave-vnbh krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:16:22.785100 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:36693
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:36693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:22.907990 7920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:22.908317 7920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:22.908360 7920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:22.912744 7920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:22.912829 7920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:22.912850 7920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:22.912868 7920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:22.912884 7920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:22.918715 7920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:36693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:36693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:22.919955 7920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:22.921034 7920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:22.930322 7928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:22.930274 7926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:22.930385 7920 server_base.cc:1034] running on GCE node
W20260926 20:16:22.930308 7925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:22.931298 7920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:22.932627 7920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:22.933861 7920 hybrid_clock.cc:648] HybridClock initialized: now 1790453782933832 us; error 43 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T20:16:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1344 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:16:23 dist-test-slave-vnbh krb5kdc[7933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:16:23 dist-test-slave-vnbh krb5kdc[7933](info): set up 2 sockets
Sep 26 20:16:23 dist-test-slave-vnbh krb5kdc[7933](info): commencing operation
krb5kdc: starting...
W20260926 20:16:23.999203 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:24 dist-test-slave-vnbh krb5kdc[7933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:16:24.125257 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:35793
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:35793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:24.241137 7946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:24.241643 7946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:24.241761 7946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:24.246176 7946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:24.246305 7946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:24.246343 7946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:24.246383 7946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:24.246418 7946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:24.252492 7946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:35793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:35793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:24.253859 7946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:24.254987 7946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:24.264525 7954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:24.264532 7951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:24.264635 7946 server_base.cc:1034] running on GCE node
W20260926 20:16:24.264528 7952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:24.265555 7946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:24.266778 7946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:24.267976 7946 hybrid_clock.cc:648] HybridClock initialized: now 1790453784267941 us; error 51 us; skew 500 ppm
Sep 26 20:16:24 dist-test-slave-vnbh krb5kdc[7933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453784, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:24.271030 7946 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:16:24.272042 7946 webserver.cc:460] Webserver started at http://127.31.119.254:37021/ using document root <none> and password file <none>
I20260926 20:16:24.272650 7946 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:24.272713 7946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:24.272887 7946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:24.275137 7946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "b69c2212871bf8c5f5bf60f514664db7"
server_key_iv: "ac0ba3f6e56893c64dfa75b8096b7268"
server_key_version: "encryptionkey@0"
I20260926 20:16:24.275686 7946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "b69c2212871bf8c5f5bf60f514664db7"
server_key_iv: "ac0ba3f6e56893c64dfa75b8096b7268"
server_key_version: "encryptionkey@0"
I20260926 20:16:24.279748 7946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 20:16:24.282646 7962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:24.284210 7946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:24.284335 7946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "b69c2212871bf8c5f5bf60f514664db7"
server_key_iv: "ac0ba3f6e56893c64dfa75b8096b7268"
server_key_version: "encryptionkey@0"
I20260926 20:16:24.284444 7946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:24.304147 7946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:24.307875 7946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:24.308037 7946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:24.315872 7946 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:35793
I20260926 20:16:24.315907 8014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:35793 every 8 connection(s)
I20260926 20:16:24.316908 7946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:24.319901 8015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:24.322679 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 7946
I20260926 20:16:24.322795 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:24.323135 32223 external_mini_cluster.cc:1428] Setting key 9cb60838ad31d2efdf954adf3e4c679d
I20260926 20:16:24.326188 8015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: Bootstrap starting.
I20260926 20:16:24.328944 8015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:24.329989 8015 log.cc:824] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:24.332701 8015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: No bootstrap required, opened a new log
Sep 26 20:16:24 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:24.336356 8015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } }
I20260926 20:16:24.336650 8015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:24.336732 8015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9a72a34ea54bd38f56e09f4a0a2c4c, State: Initialized, Role: FOLLOWER
I20260926 20:16:24.337209 8015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } }
I20260926 20:16:24.337409 8015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:24.337495 8015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:24.337610 8015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:24.338917 8015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } }
I20260926 20:16:24.339351 8015 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f9a72a34ea54bd38f56e09f4a0a2c4c; no voters:
I20260926 20:16:24.339737 8015 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:24.339864 8020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:24.340114 8020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 1 LEADER]: Becoming Leader. State: Replica: 3f9a72a34ea54bd38f56e09f4a0a2c4c, State: Running, Role: LEADER
I20260926 20:16:24.340564 8020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } }
I20260926 20:16:24.340858 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:24.324779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.31.119.254:35793)
0926 20:16:24.325250 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:24.325268 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:24.325376 (+ 108us) server_negotiation.cc:407] Connection header received
0926 20:16:24.326026 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:24.326045 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:24.326465 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:24.326768 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:24.327813 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:24.329014 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:24.330314 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:24.331048 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:24.331198 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:24.334232 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:24.334238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:24.334254 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:24.334286 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:24.337202 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:24.337825 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:24.337834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:24.337843 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:24.338007 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:24.338294 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:24.338298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:24.338299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:24.338911 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:24.339129 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:24.339453 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 20:16:24.339816 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":179,"threads_started":1}
I20260926 20:16:24.341110 8015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:24.343673 8021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } } }
I20260926 20:16:24.343815 8021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:24.344231 8022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9a72a34ea54bd38f56e09f4a0a2c4c. Latest consensus state: current_term: 1 leader_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } } }
I20260926 20:16:24.344355 8022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [sys.catalog]: This master's current role is: LEADER
W20260926 20:16:24.346107 8034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:16:24.346207 8034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:16:24.346307 8036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:24.347187 8036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:24.353494 8036 catalog_manager.cc:1347] Generated new cluster ID: 85ba7b1869444a459c190f90eb49b313
I20260926 20:16:24.353590 8036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:24.368502 8036 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:24.369637 8036 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:24.390868 8036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: Generated new TSK 0
I20260926 20:16:24.391682 8036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:16:24.422746 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35793
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:24.568395 8041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:24.568737 8041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:24.568817 8041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:24.573211 8041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:24.573359 8041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:24.573472 8041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:24.579260 8041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:24.580476 8041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:24.581754 8041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:24.593003 8041 server_base.cc:1034] running on GCE node
W20260926 20:16:24.592893 8046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:24.592913 8049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:24.592902 8047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:24.593670 8041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:24.594396 8041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:24.595626 8041 hybrid_clock.cc:648] HybridClock initialized: now 1790453784595598 us; error 38 us; skew 500 ppm
Sep 26 20:16:24 dist-test-slave-vnbh krb5kdc[7933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453784, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:24.598946 8041 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:16:24.600013 8041 webserver.cc:460] Webserver started at http://127.31.119.193:46047/ using document root <none> and password file <none>
I20260926 20:16:24.600666 8041 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:24.600733 8041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:24.600997 8041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:24.603271 8041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "8c60435b1a464944a4da2a667e7b9c2a"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "7ddff3c26a00ca67860d043d04a6c8fc"
server_key_iv: "f72593f946b4d04c4e3128342699f078"
server_key_version: "encryptionkey@0"
I20260926 20:16:24.603852 8041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "8c60435b1a464944a4da2a667e7b9c2a"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "7ddff3c26a00ca67860d043d04a6c8fc"
server_key_iv: "f72593f946b4d04c4e3128342699f078"
server_key_version: "encryptionkey@0"
I20260926 20:16:24.607970 8041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:16:24.610813 8056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:24.612346 8041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:16:24.612480 8041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "8c60435b1a464944a4da2a667e7b9c2a"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "7ddff3c26a00ca67860d043d04a6c8fc"
server_key_iv: "f72593f946b4d04c4e3128342699f078"
server_key_version: "encryptionkey@0"
I20260926 20:16:24.612756 8041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:24.626816 8041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:24.630759 8041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:24.630975 8041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:24.631628 8041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:24.632541 8041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:24.632618 8041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:24.632690 8041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:24.632741 8041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:24.642258 8041 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:35595
I20260926 20:16:24.642325 8169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:35595 every 8 connection(s)
I20260926 20:16:24.643348 8041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:24.650890 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 8041
I20260926 20:16:24.651006 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:16:24.651314 32223 external_mini_cluster.cc:1428] Setting key 57f5d9e8402ae04dac272e172e8ce2d6
Sep 26 20:16:24 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453784, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:24.658557 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:24.645401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:59679 (local address 127.31.119.254:35793)
0926 20:16:24.645556 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:24.645561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:24.646509 (+ 948us) server_negotiation.cc:407] Connection header received
0926 20:16:24.647916 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:24.647919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:24.647969 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:24.648054 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:24.649403 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:24.649937 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:24.651163 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:24.651432 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:24.651558 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:24.654619 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:24.654623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:24.654625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:24.654650 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:24.656234 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:24.656840 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:24.656849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:24.656850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:24.656940 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:24.657375 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:24.657378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:24.657380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:24.657582 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:24.657693 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:24.658247 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 20:16:24.658388 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:16:24.659474 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:24.645879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:59679)
0926 20:16:24.646351 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:24.646397 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:24.647717 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:24.648229 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:24.648237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:24.648711 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:24.649217 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:24.649231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:24.650063 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:24.650066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:24.651022 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:24.651037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:24.651612 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:24.651616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:24.651786 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:24.652442 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:24.652460 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:24.654417 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:24.656373 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:24.656379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:24.656400 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:24.656720 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:24.657059 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:24.657063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:24.657064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:24.657228 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:24.657693 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:24.657702 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:24.657994 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 20:16:24.658194 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 20:16:24.658577 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
I20260926 20:16:24.661072 8170 heartbeater.cc:347] Connected to a master server at 127.31.119.254:35793
I20260926 20:16:24.661407 8170 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:24.661926 8170 heartbeater.cc:511] Master 127.31.119.254:35793 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
I20260926 20:16:24.663844 7979 ts_manager.cc:194] Registered new tserver with Master: 8c60435b1a464944a4da2a667e7b9c2a (127.31.119.193:35595)
I20260926 20:16:24.665045 7979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:59679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:16:24.681018 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35793
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:24.801859 8175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:24.802168 8175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:24.802213 8175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:24.806392 8175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:24.806466 8175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:24.806530 8175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:24.812091 8175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:24.813194 8175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:24.814283 8175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:24.825587 8183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:24.825608 8180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:24.825610 8181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:24.825878 8175 server_base.cc:1034] running on GCE node
I20260926 20:16:24.826375 8175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:24.827112 8175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:24.828287 8175 hybrid_clock.cc:648] HybridClock initialized: now 1790453784828223 us; error 96 us; skew 500 ppm
Sep 26 20:16:24 dist-test-slave-vnbh krb5kdc[7933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453784, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:24.831565 8175 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:16:24.832639 8175 webserver.cc:460] Webserver started at http://127.31.119.194:40841/ using document root <none> and password file <none>
I20260926 20:16:24.833382 8175 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:24.833461 8175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:24.833698 8175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:24.836093 8175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "a4d51e201b0c413ca01e4208cedde928"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "25d2f6556d89ec2dc603d16e56e69ed6"
server_key_iv: "336afd594881adf65dc200fe4339d517"
server_key_version: "encryptionkey@0"
I20260926 20:16:24.836745 8175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "a4d51e201b0c413ca01e4208cedde928"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "25d2f6556d89ec2dc603d16e56e69ed6"
server_key_iv: "336afd594881adf65dc200fe4339d517"
server_key_version: "encryptionkey@0"
I20260926 20:16:24.841372 8175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 20:16:24.844732 8190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:24.846346 8175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:24.846535 8175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "a4d51e201b0c413ca01e4208cedde928"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "25d2f6556d89ec2dc603d16e56e69ed6"
server_key_iv: "336afd594881adf65dc200fe4339d517"
server_key_version: "encryptionkey@0"
I20260926 20:16:24.846676 8175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:24.862355 8175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:24.866084 8175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:24.866305 8175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:24.866940 8175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:24.867823 8175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:24.867898 8175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:24.867974 8175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:24.868036 8175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:24.878379 8175 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:43477
I20260926 20:16:24.878947 8303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:43477 every 8 connection(s)
I20260926 20:16:24.879595 8175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:24.888773 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 8175
I20260926 20:16:24.888967 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:24.889267 32223 external_mini_cluster.cc:1428] Setting key 0ff8dc7f47a3c607ec29fb447cccb4fc
Sep 26 20:16:24 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453784, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:24.895008 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:24.881764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:37471 (local address 127.31.119.254:35793)
0926 20:16:24.881916 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:24.881921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:24.882871 (+ 950us) server_negotiation.cc:407] Connection header received
0926 20:16:24.884312 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:24.884315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:24.884375 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:24.884486 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:24.885798 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:24.886342 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:24.887560 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:24.887804 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:24.887921 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:24.891087 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:24.891090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:24.891093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:24.891126 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:24.892707 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:24.893332 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:24.893339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:24.893344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:24.893435 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:24.893811 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:24.893815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:24.893817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:24.894029 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:24.894141 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:24.894692 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 20:16:24.894826 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:16:24.895942 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:24.882192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:37471)
0926 20:16:24.882702 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:24.882750 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:24.884074 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:24.884681 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:24.884689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:24.885084 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:24.885636 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:24.885650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:24.886485 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:24.886493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:24.887419 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:24.887429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:24.887981 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:24.887986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:24.888171 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:24.888956 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:24.888984 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:24.890898 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:24.892880 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:24.892887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:24.892906 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:24.893206 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:24.893545 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:24.893548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:24.893550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:24.893712 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:24.894140 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:24.894148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:24.894432 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 20:16:24.894651 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 20:16:24.895027 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
I20260926 20:16:24.897416 8304 heartbeater.cc:347] Connected to a master server at 127.31.119.254:35793
I20260926 20:16:24.897740 8304 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:24.898453 8304 heartbeater.cc:511] Master 127.31.119.254:35793 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
I20260926 20:16:24.899915 7979 ts_manager.cc:194] Registered new tserver with Master: a4d51e201b0c413ca01e4208cedde928 (127.31.119.194:43477)
I20260926 20:16:24.900631 7979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:37471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:16:24.921500 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35793
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:25.038520 8309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:25.038807 8309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:25.038851 8309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:25.043320 8309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:25.043402 8309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:25.043498 8309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:25.049057 8309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:25.050397 8309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:25.051465 8309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:25.061692 8314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:25.061730 8317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:25.061695 8315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:25.061791 8309 server_base.cc:1034] running on GCE node
I20260926 20:16:25.062412 8309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:25.063092 8309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:25.064296 8309 hybrid_clock.cc:648] HybridClock initialized: now 1790453785064273 us; error 33 us; skew 500 ppm
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:25.067700 8309 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:16:25.068719 8309 webserver.cc:460] Webserver started at http://127.31.119.195:46329/ using document root <none> and password file <none>
I20260926 20:16:25.069429 8309 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:25.069504 8309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:25.069728 8309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:25.071973 8309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "cbd9621d29a44acf89c4260f227a68d7"
format_stamp: "Formatted at 2026-09-26 20:16:25 on dist-test-slave-vnbh"
server_key: "a6bd68048762d5ec0d55f063118641bf"
server_key_iv: "db0f0ce793de05c76b6851345aad7e02"
server_key_version: "encryptionkey@0"
I20260926 20:16:25.072597 8309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "cbd9621d29a44acf89c4260f227a68d7"
format_stamp: "Formatted at 2026-09-26 20:16:25 on dist-test-slave-vnbh"
server_key: "a6bd68048762d5ec0d55f063118641bf"
server_key_iv: "db0f0ce793de05c76b6851345aad7e02"
server_key_version: "encryptionkey@0"
I20260926 20:16:25.076719 8309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:16:25.079641 8324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:25.081105 8309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:25.081223 8309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "cbd9621d29a44acf89c4260f227a68d7"
format_stamp: "Formatted at 2026-09-26 20:16:25 on dist-test-slave-vnbh"
server_key: "a6bd68048762d5ec0d55f063118641bf"
server_key_iv: "db0f0ce793de05c76b6851345aad7e02"
server_key_version: "encryptionkey@0"
I20260926 20:16:25.081441 8309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:25.099042 8309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:25.102831 8309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:25.103055 8309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:25.103713 8309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:25.104604 8309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:25.104651 8309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:25.104718 8309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:25.104755 8309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:25.115104 8309 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:37765
I20260926 20:16:25.115126 8437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:37765 every 8 connection(s)
I20260926 20:16:25.116232 8309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:25.119134 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 8309
I20260926 20:16:25.119290 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:25.119617 32223 external_mini_cluster.cc:1428] Setting key 8c97422ead48ffc6277fda493bac6b95
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:25.131820 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:25.118539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:53845 (local address 127.31.119.254:35793)
0926 20:16:25.118691 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:25.118695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:25.119807 (+ 1112us) server_negotiation.cc:407] Connection header received
0926 20:16:25.121516 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:25.121520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:25.121590 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:25.121699 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:25.122934 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.123482 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.124721 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.124949 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.125064 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.127826 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:25.127829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:25.127831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:25.127864 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:25.129547 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:25.130124 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:25.130129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:25.130131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:25.130217 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:25.130628 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:25.130633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:25.130635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:25.130842 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:25.130960 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:25.131512 (+ 552us) server_negotiation.cc:299] Negotiation successful
0926 20:16:25.131652 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:16:25.132974 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:25.118969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:53845)
0926 20:16:25.119609 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:25.119683 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:25.121245 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:25.121888 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:25.121896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:25.122274 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:25.122780 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.122793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.123616 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.123624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.124579 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.124590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.125111 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.125115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.125370 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.126036 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:25.126053 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:25.127623 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:25.129682 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:25.129690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:25.129709 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:25.130007 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:25.130324 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:25.130327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:25.130329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:25.130538 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:25.130959 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:25.130968 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:25.131259 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 20:16:25.131473 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 20:16:25.131835 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":180,"threads_started":1}
I20260926 20:16:25.134513 8438 heartbeater.cc:347] Connected to a master server at 127.31.119.254:35793
I20260926 20:16:25.135033 8438 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:25.135634 8438 heartbeater.cc:511] Master 127.31.119.254:35793 requested a full tablet report, sending...
I20260926 20:16:25.136977 7979 ts_manager.cc:194] Registered new tserver with Master: cbd9621d29a44acf89c4260f227a68d7 (127.31.119.195:37765)
I20260926 20:16:25.137689 7979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:53845
I20260926 20:16:25.146340 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab.
Entry for principal oryx/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab.
WARNING: no policy specified for oryx/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab.
Entry for principal oryx/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab.
WARNING: no policy specified for oryx/127.31.119.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.119.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab.
Entry for principal oryx/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab.
I20260926 20:16:25.232617 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 8041
I20260926 20:16:25.238487 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 8175
I20260926 20:16:25.244169 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 8309
I20260926 20:16:25.249752 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 7946
I20260926 20:16:25.256390 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:35793
--webserver_interface=127.31.119.254
--webserver_port=37021
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:35793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:25.372838 8447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:25.373181 8447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:25.373227 8447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:25.377513 8447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:25.377596 8447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:25.377617 8447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:25.377635 8447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:25.377650 8447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:25.383314 8447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:35793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:35793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=37021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:25.384567 8447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:25.385703 8447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:25.394451 8452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:25.394451 8455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:25.394759 8453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:25.394623 8447 server_base.cc:1034] running on GCE node
I20260926 20:16:25.395453 8447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:25.396745 8447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:25.397966 8447 hybrid_clock.cc:648] HybridClock initialized: now 1790453785397946 us; error 37 us; skew 500 ppm
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:25.401160 8447 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:16:25.402261 8447 webserver.cc:460] Webserver started at http://127.31.119.254:37021/ using document root <none> and password file <none>
I20260926 20:16:25.402901 8447 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:25.403031 8447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:25.407418 8447 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:16:25.410163 8462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:25.411978 8447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:25.412151 8447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "b69c2212871bf8c5f5bf60f514664db7"
server_key_iv: "ac0ba3f6e56893c64dfa75b8096b7268"
server_key_version: "encryptionkey@0"
I20260926 20:16:25.413031 8447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:25.442672 8447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:25.446482 8447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:25.446655 8447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:25.454221 8447 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:35793
I20260926 20:16:25.454236 8514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:35793 every 8 connection(s)
I20260926 20:16:25.455349 8447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:25.459897 8515 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:16:25.461764 8515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: Bootstrap starting.
I20260926 20:16:25.463792 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 8447
I20260926 20:16:25.464313 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:35595
--local_ip_for_outbound_sockets=127.31.119.193
--tserver_master_addrs=127.31.119.254:35793
--webserver_port=46047
--webserver_interface=127.31.119.193
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.119.193
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:16:25.479579 8515 log.cc:824] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:25.486816 8515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:16:25.487247 8515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: Bootstrap complete.
I20260926 20:16:25.491976 8515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } }
I20260926 20:16:25.492394 8515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9a72a34ea54bd38f56e09f4a0a2c4c, State: Initialized, Role: FOLLOWER
I20260926 20:16:25.492897 8515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } }
I20260926 20:16:25.493011 8515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:25.493110 8515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:25.493206 8515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:16:25.495198 8515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } }
I20260926 20:16:25.495553 8515 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f9a72a34ea54bd38f56e09f4a0a2c4c; no voters:
I20260926 20:16:25.495905 8515 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:16:25.495999 8519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:16:25.496194 8519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [term 2 LEADER]: Becoming Leader. State: Replica: 3f9a72a34ea54bd38f56e09f4a0a2c4c, State: Running, Role: LEADER
I20260926 20:16:25.496495 8519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } }
I20260926 20:16:25.497197 8515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:25.497476 8519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9a72a34ea54bd38f56e09f4a0a2c4c. Latest consensus state: current_term: 2 leader_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } } }
I20260926 20:16:25.497611 8519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:25.497982 8520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9a72a34ea54bd38f56e09f4a0a2c4c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 35793 } } }
I20260926 20:16:25.498157 8520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:25.499310 8523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:25.503074 8523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:25.503734 8523 catalog_manager.cc:1259] Loaded cluster ID: 85ba7b1869444a459c190f90eb49b313
I20260926 20:16:25.503800 8523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:25.506196 8523 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:25.506882 8523 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f9a72a34ea54bd38f56e09f4a0a2c4c: Loaded TSK: 0
I20260926 20:16:25.508176 8523 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:16:25.589072 8517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:25.589406 8517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:25.589495 8517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:25.594348 8517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:25.594427 8517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:25.594532 8517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:25.600540 8517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.193.keytab
--principal=oryx/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:35595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=46047
--tserver_master_addrs=127.31.119.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:25.601756 8517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:25.602842 8517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:25.614583 8541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:25.614643 8542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:25.614607 8544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:25.615201 8517 server_base.cc:1034] running on GCE node
I20260926 20:16:25.615724 8517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:25.616544 8517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:25.617805 8517 hybrid_clock.cc:648] HybridClock initialized: now 1790453785617767 us; error 52 us; skew 500 ppm
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453785, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:25.621722 8517 init.cc:377] Logged in from keytab as oryx/127.31.119.193@KRBTEST.COM (short username oryx)
I20260926 20:16:25.622918 8517 webserver.cc:460] Webserver started at http://127.31.119.193:46047/ using document root <none> and password file <none>
I20260926 20:16:25.623572 8517 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:25.623693 8517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:25.628059 8517 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:16:25.630402 8551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:25.631865 8517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:25.632004 8517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "8c60435b1a464944a4da2a667e7b9c2a"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "7ddff3c26a00ca67860d043d04a6c8fc"
server_key_iv: "f72593f946b4d04c4e3128342699f078"
server_key_version: "encryptionkey@0"
I20260926 20:16:25.632597 8517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:25.647002 8517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:25.650941 8517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:25.651168 8517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:25.651846 8517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:25.652773 8517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:25.652824 8517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:25.652866 8517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:25.652923 8517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:25.662762 8517 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:35595
I20260926 20:16:25.662784 8664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:35595 every 8 connection(s)
I20260926 20:16:25.663931 8517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:25.672118 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 8517
I20260926 20:16:25.672638 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:43477
--local_ip_for_outbound_sockets=127.31.119.194
--tserver_master_addrs=127.31.119.254:35793
--webserver_port=40841
--webserver_interface=127.31.119.194
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.119.194
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:25.679061 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:25.666498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:60705)
0926 20:16:25.666965 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:25.667031 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:25.668346 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:25.669627 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:25.669634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:25.670019 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:25.670527 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.670542 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.672002 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.672004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.673131 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.673142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.674371 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.674376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.674690 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.675401 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:25.675419 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:25.678223 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":147,"threads_started":1}
W20260926 20:16:25.679728 8665 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:35793 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.119.254:35793: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
W20260926 20:16:25.679809 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:25.666362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:60705 (local address 127.31.119.254:35793)
0926 20:16:25.666884 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:25.666935 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:25.667156 (+ 221us) server_negotiation.cc:407] Connection header received
0926 20:16:25.668789 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:25.668800 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:25.669171 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:25.669496 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:25.670750 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.671839 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.673258 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.674161 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.674300 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.678678 (+ 4378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:60705: BlockingRecv error: recv got EOF from 127.31.119.193:60705 (error 108)
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":146,"threads_started":1}
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:25.685922 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:25.680367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:50909)
0926 20:16:25.680471 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:25.680487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:25.680604 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:25.680843 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:25.680845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:25.681014 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:25.681106 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.681111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.681931 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.681936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.682488 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.682496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.682977 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.682981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.683074 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.683717 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:25.683730 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:25.685748 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 20:16:25.686066 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:25.680482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:50909 (local address 127.31.119.254:35793)
0926 20:16:25.680613 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:25.680618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:25.680634 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:16:25.680683 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:25.680686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:25.680732 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:25.680829 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:25.681222 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.681803 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.682608 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.682855 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.682977 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.685943 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:50909: BlockingRecv error: recv got EOF from 127.31.119.193:50909 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:25.692653 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:25.686937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:54423)
0926 20:16:25.687050 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:25.687065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:25.687146 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:25.687393 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:25.687395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:25.687574 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:25.687668 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.687674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.688496 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.688499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.689403 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.689413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.689870 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.689873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.689960 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.690502 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:25.690518 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:25.692513 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 20:16:25.692839 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:25.686986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:54423 (local address 127.31.119.254:35793)
0926 20:16:25.687099 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:25.687104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:25.687126 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:16:25.687238 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:25.687240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:25.687281 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:25.687349 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:25.687803 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.688330 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.689531 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.689760 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.689848 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.692721 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:54423: BlockingRecv error: recv got EOF from 127.31.119.193:54423 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 20:16:25.693004 8665 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:16:25.793753 8669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:25.794078 8669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:25.794159 8669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:25.798470 8669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:25.798560 8669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:25.798677 8669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:25.804265 8669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.194.keytab
--principal=oryx/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:43477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=40841
--tserver_master_addrs=127.31.119.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:25.805573 8669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:25.806694 8669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:25.817335 8675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:25.817256 8674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:25.817353 8677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:25.817556 8669 server_base.cc:1034] running on GCE node
I20260926 20:16:25.818346 8669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:25.819144 8669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:25.820369 8669 hybrid_clock.cc:648] HybridClock initialized: now 1790453785820341 us; error 50 us; skew 500 ppm
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453785, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:25.824275 8669 init.cc:377] Logged in from keytab as oryx/127.31.119.194@KRBTEST.COM (short username oryx)
I20260926 20:16:25.825498 8669 webserver.cc:460] Webserver started at http://127.31.119.194:40841/ using document root <none> and password file <none>
I20260926 20:16:25.826210 8669 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:25.826296 8669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:25.831588 8669 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 20:16:25.833895 8684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:25.835350 8669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:25.835544 8669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "a4d51e201b0c413ca01e4208cedde928"
format_stamp: "Formatted at 2026-09-26 20:16:24 on dist-test-slave-vnbh"
server_key: "25d2f6556d89ec2dc603d16e56e69ed6"
server_key_iv: "336afd594881adf65dc200fe4339d517"
server_key_version: "encryptionkey@0"
I20260926 20:16:25.836297 8669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:25.863420 8669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:25.867515 8669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:25.867764 8669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:25.868448 8669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:25.869371 8669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:25.869488 8669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:25.869565 8669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:25.869603 8669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:25.879441 8669 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:43477
I20260926 20:16:25.879619 8797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:43477 every 8 connection(s)
I20260926 20:16:25.880582 8669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:25.891086 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 8669
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:25.891615 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:37765
--local_ip_for_outbound_sockets=127.31.119.195
--tserver_master_addrs=127.31.119.254:35793
--webserver_port=46329
--webserver_interface=127.31.119.195
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.119.195
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:16:25.893335 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:25.882724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:40769 (local address 127.31.119.254:35793)
0926 20:16:25.882874 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:25.882878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:25.883829 (+ 951us) server_negotiation.cc:407] Connection header received
0926 20:16:25.885220 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:25.885224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:25.885284 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:25.885466 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:25.886812 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.887354 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.888745 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.888976 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.889081 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.893179 (+ 4098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:40769: BlockingRecv error: recv got EOF from 127.31.119.194:40769 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:16:25.894025 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:25.883089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:40769)
0926 20:16:25.883648 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:25.883694 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:25.884943 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:25.885701 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:25.885710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:25.886118 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:25.886649 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.886662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.887589 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.887592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.888593 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.888602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.889143 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.889145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.889331 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.890162 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:25.890180 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:25.893002 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
W20260926 20:16:25.894515 8798 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:35793 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.119.254:35793: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:25.902311 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:25.895013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:55991)
0926 20:16:25.895217 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:25.895234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:25.895332 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:25.895674 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:25.895680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:25.895916 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:25.896084 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.896095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.897013 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.897018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.898153 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.898168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.898710 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.898717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.898848 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.899575 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:25.899594 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:25.902106 (+ 2512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260926 20:16:25.902709 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:25.895108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:55991 (local address 127.31.119.254:35793)
0926 20:16:25.895239 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:25.895243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:25.895281 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:16:25.895429 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:25.895431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:25.895536 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:25.895613 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:25.896235 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.896892 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.898304 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.898574 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.898673 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.902271 (+ 3598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:55991: BlockingRecv error: recv got EOF from 127.31.119.194:55991 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:25 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:25.909076 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:25.902944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:42411)
0926 20:16:25.903091 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:25.903104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:25.903206 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:25.903677 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:25.903679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:25.903819 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:25.903907 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.903913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.904997 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.905001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.905724 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:25.905735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.906352 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.906355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:25.906419 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.907008 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:25.907021 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:25.908930 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 20:16:25.909449 8798 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:16:25.909945 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:25.903232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:42411 (local address 127.31.119.254:35793)
0926 20:16:25.903406 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:25.903413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:25.903430 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:16:25.903479 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:25.903484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:25.903546 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:25.903648 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:25.904022 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.904852 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.905843 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:25.906216 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:25.906315 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:25.909796 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:42411: BlockingRecv error: recv got EOF from 127.31.119.194:42411 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 20:16:26.013178 8801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:26.013584 8801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:26.013674 8801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:26.018621 8801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:26.018728 8801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:26.018832 8801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:26.024822 8801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/krb5kdc/oryx_127.31.119.195.keytab
--principal=oryx/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:37765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=46329
--tserver_master_addrs=127.31.119.254:35793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:26.026146 8801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:26.027259 8801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:16:26.038136 8801 server_base.cc:1034] running on GCE node
W20260926 20:16:26.038064 8807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:26.038058 8806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:26.038053 8809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:26.038739 8801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:26.039534 8801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:26.040763 8801 hybrid_clock.cc:648] HybridClock initialized: now 1790453786040732 us; error 47 us; skew 500 ppm
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453786, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:26.043933 8801 init.cc:377] Logged in from keytab as oryx/127.31.119.195@KRBTEST.COM (short username oryx)
I20260926 20:16:26.044924 8801 webserver.cc:460] Webserver started at http://127.31.119.195:46329/ using document root <none> and password file <none>
I20260926 20:16:26.045782 8801 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:26.045899 8801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:26.050171 8801 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:16:26.052441 8816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:26.053834 8801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:26.053977 8801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "cbd9621d29a44acf89c4260f227a68d7"
format_stamp: "Formatted at 2026-09-26 20:16:25 on dist-test-slave-vnbh"
server_key: "a6bd68048762d5ec0d55f063118641bf"
server_key_iv: "db0f0ce793de05c76b6851345aad7e02"
server_key_version: "encryptionkey@0"
I20260926 20:16:26.054561 8801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:26.068521 8801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:26.072067 8801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:26.072320 8801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:26.073156 8801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:26.074151 8801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:26.074231 8801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:26.074301 8801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:26.074347 8801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:26.084733 8801 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:37765
I20260926 20:16:26.084803 8929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:37765 every 8 connection(s)
I20260926 20:16:26.085969 8801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:26.089537 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 8801
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:16:26.098456 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:26.088055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:37153 (local address 127.31.119.254:35793)
0926 20:16:26.088252 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:26.088257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:26.089473 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 20:16:26.091164 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:26.091167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:26.091240 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:26.091325 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:26.092603 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.093134 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.094421 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.094764 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.094894 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.098324 (+ 3430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:37153: BlockingRecv error: recv got EOF from 127.31.119.195:37153 (error 108)
Metrics: {"server-negotiator.queue_time_us":69,"spinlock_wait_cycles":896}
I20260926 20:16:26.099202 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:26.088661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:37153)
0926 20:16:26.089223 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:26.089270 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:26.090942 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:26.091525 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:26.091533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:26.091903 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:26.092435 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:26.092450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.093265 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.093271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:26.094283 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:26.094295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.094935 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.094938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:26.095106 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.095674 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:26.095690 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:26.098222 (+ 2532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":137,"threads_started":1}
W20260926 20:16:26.099740 8930 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:35793 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.119.254:35793: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
I20260926 20:16:26.100497 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:26.090700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.31.119.254:35793)
0926 20:16:26.090960 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:26.090963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:26.090974 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:16:26.091013 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:26.091016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:26.091055 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:26.091163 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:26.091921 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.092475 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.093482 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.093723 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.093835 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.095525 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:26.095531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:26.095546 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:26.095573 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:26.097696 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:26.098163 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:26.098170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:26.098176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:26.098310 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:26.098701 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:26.098704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:26.098706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:26.099113 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:26.099324 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:26.100112 (+ 788us) server_negotiation.cc:299] Negotiation successful
0926 20:16:26.100370 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":113,"threads_started":1}
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:26.105862 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:26.100222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:54313)
0926 20:16:26.100450 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:26.100463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:26.100638 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:26.100914 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:26.100917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:26.101083 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:26.101191 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:26.101197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.102012 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.102015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:26.102793 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:26.102804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.103281 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.103286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:26.103384 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.103806 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:26.103817 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:26.105762 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159}
W20260926 20:16:26.105983 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:26.100306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:54313 (local address 127.31.119.254:35793)
0926 20:16:26.100425 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:26.100429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:26.100504 (+ 75us) server_negotiation.cc:407] Connection header received
0926 20:16:26.100749 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:26.100752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:26.100800 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:26.100876 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:26.101371 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.101893 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.102912 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.103170 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.103252 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.105897 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:54313: BlockingRecv error: recv got EOF from 127.31.119.195:54313 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:26.111521 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:26.106550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:44111)
0926 20:16:26.106689 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:26.106701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:26.106771 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:26.106988 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:26.106990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:26.107118 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:26.107197 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:26.107202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.107876 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.107879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:26.108440 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:26.108449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.108906 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.108914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:26.108978 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.109465 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:26.109522 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:26.111416 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 20:16:26.111685 8667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:26.106662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:44111 (local address 127.31.119.254:35793)
0926 20:16:26.106808 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:26.106811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:26.106822 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:16:26.106854 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:26.106857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:26.106890 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:26.106955 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:26.107294 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.107783 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.108587 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.108786 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.108863 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.111555 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:44111: BlockingRecv error: recv got EOF from 127.31.119.195:44111 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 20:16:26.111832 8930 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:26.703879 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:26.693696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:46523)
0926 20:16:26.694042 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:26.694056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:26.694152 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:26.694407 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:26.694409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:26.694636 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:26.694731 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:26.694737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.695517 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.695520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:26.696120 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:26.696129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.696650 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.696654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:26.696724 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.701186 (+ 4462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:26.701204 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:26.703746 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":91,"threads_started":1}
W20260926 20:16:26.704062 8936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:26.693844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:46523 (local address 127.31.119.254:35793)
0926 20:16:26.694110 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:26.694113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:26.694127 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:16:26.694242 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:26.694244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:26.694287 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:26.694365 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:26.694844 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.695379 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.696228 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.696450 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.696584 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.703913 (+ 7329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:46523: BlockingRecv error: recv got EOF from 127.31.119.193:46523 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":72,"threads_started":1}
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:26 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:26.916694 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:26.910324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:45243)
0926 20:16:26.910622 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:26.910636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:26.910741 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:26.911012 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:26.911013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:26.911193 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:26.911289 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:26.911294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.912140 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.912147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:26.912786 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:26.912797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.913541 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.913545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:26.913631 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.914313 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:26.914331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:26.916482 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":68,"threads_started":1}
W20260926 20:16:26.916769 8936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:26.910409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:45243 (local address 127.31.119.254:35793)
0926 20:16:26.910571 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:26.910576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:26.910754 (+ 178us) server_negotiation.cc:407] Connection header received
0926 20:16:26.910810 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:26.910813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:26.910893 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:26.910977 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:26.911439 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.911991 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.912927 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:26.913385 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:26.913542 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:26.916644 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:45243: BlockingRecv error: recv got EOF from 127.31.119.194:45243 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 20:16:27 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:27 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:27.118888 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:27.112587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:57093)
0926 20:16:27.112969 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:27.112989 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:27.113140 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:27.113525 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:27.113530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:27.113725 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:27.113816 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:27.113821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.114615 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.114619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:27.115222 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:27.115230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.115864 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.115868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:27.115956 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:27.116617 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:27.116630 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:27.118677 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":89,"threads_started":1}
W20260926 20:16:27.118928 8936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:27.112730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:57093 (local address 127.31.119.254:35793)
0926 20:16:27.112924 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:27.112929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:27.113070 (+ 141us) server_negotiation.cc:407] Connection header received
0926 20:16:27.113249 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:27.113252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:27.113341 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:27.113436 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:27.113970 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.114489 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.115372 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.115720 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.115864 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:27.118818 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:57093: BlockingRecv error: recv got EOF from 127.31.119.195:57093 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 20:16:27 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:27 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:27.711038 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:27.704745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:33259)
0926 20:16:27.705094 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:27.705109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:27.705216 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:27.705598 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:27.705600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:27.705795 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:27.705896 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:27.705902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.706719 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.706721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:27.707330 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:27.707338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.707813 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.707816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:27.707884 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:27.708540 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:27.708553 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:27.710832 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
W20260926 20:16:27.711122 8940 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:27.704895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:33259 (local address 127.31.119.254:35793)
0926 20:16:27.705131 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:27.705134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:27.705165 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:16:27.705387 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:27.705390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:27.705440 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:27.705526 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:27.706053 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.706607 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.707438 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.707705 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.707788 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:27.710972 (+ 3184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:33259: BlockingRecv error: recv got EOF from 127.31.119.193:33259 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":57,"threads_started":1}
Sep 26 20:16:27 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:27 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:27.923640 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:27.917668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:42099)
0926 20:16:27.917982 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:27.918001 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:27.918247 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:27.918584 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:27.918587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:27.918790 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:27.918883 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:27.918890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.919662 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.919667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:27.920224 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:27.920232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.920723 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.920727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:27.920812 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:27.921468 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:27.921480 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:27.923437 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":102,"threads_started":1}
W20260926 20:16:27.923697 8940 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:27.917737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:42099 (local address 127.31.119.254:35793)
0926 20:16:27.917884 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:27.917889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:27.918175 (+ 286us) server_negotiation.cc:407] Connection header received
0926 20:16:27.918379 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:27.918383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:27.918443 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:27.918550 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:27.919009 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.919546 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.920372 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:27.920604 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:27.920720 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:27.923582 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:42099: BlockingRecv error: recv got EOF from 127.31.119.194:42099 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 20:16:28 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:28 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:28.126104 8942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:28.119831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:57877)
0926 20:16:28.120182 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:28.120199 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:28.120362 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:28.120726 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:28.120729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:28.120976 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:28.121126 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:28.121133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.122024 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.122028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:28.122683 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:28.122700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.123170 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.123174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:28.123257 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:28.123901 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:28.123914 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:28.125962 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
W20260926 20:16:28.126233 8940 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:28.120022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:57877 (local address 127.31.119.254:35793)
0926 20:16:28.120183 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:28.120189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:28.120294 (+ 105us) server_negotiation.cc:407] Connection header received
0926 20:16:28.120479 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:28.120483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:28.120597 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:28.120697 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:28.121265 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.121862 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.122809 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.123051 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.123167 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:28.126111 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:57877: BlockingRecv error: recv got EOF from 127.31.119.195:57877 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 20:16:28 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:28 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:28.717898 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:28.711998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:39805)
0926 20:16:28.712330 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:28.712344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:28.712440 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:28.712715 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:28.712718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:28.712894 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:28.712989 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:28.713022 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.713922 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.713925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:28.714507 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:28.714515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.715017 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.715019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:28.715099 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:28.715719 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:28.715731 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:28.717723 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
W20260926 20:16:28.718098 8944 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:28.712134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:39805 (local address 127.31.119.254:35793)
0926 20:16:28.712389 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:28.712394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:28.712409 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:28.712538 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:28.712540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:28.712588 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:28.712679 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:28.713130 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.713808 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.714637 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.714892 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.714988 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:28.717955 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:39805: BlockingRecv error: recv got EOF from 127.31.119.193:39805 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":63,"threads_started":1}
Sep 26 20:16:28 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:28 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:16:28.930527 8944 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:28.924624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:37011 (local address 127.31.119.254:35793)
0926 20:16:28.924811 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:28.924815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:28.924948 (+ 133us) server_negotiation.cc:407] Connection header received
0926 20:16:28.925144 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:28.925148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:28.925204 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:28.925338 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:28.925867 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.926434 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.927260 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.927493 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.927610 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:28.930415 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:37011: BlockingRecv error: recv got EOF from 127.31.119.194:37011 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 20:16:28.930471 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:28.924478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:37011)
0926 20:16:28.924849 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:28.924871 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:28.925012 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:28.925366 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:28.925369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:28.925640 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:28.925737 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:28.925743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.926560 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.926567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:28.927135 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:28.927143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:28.927612 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:28.927616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:28.927700 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:28.928294 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:28.928306 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:28.930268 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
Sep 26 20:16:29 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:29 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:29.133481 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:29.127223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:43889)
0926 20:16:29.127696 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:29.127709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:29.127848 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:29.128198 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:29.128219 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:29.128460 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:29.128574 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:29.128580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.129352 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.129354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:29.129927 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:29.129935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.130372 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.130375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:29.130435 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:29.130954 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:29.130966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:29.133174 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":114,"threads_started":1}
W20260926 20:16:29.133641 8944 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:29.127329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:43889 (local address 127.31.119.254:35793)
0926 20:16:29.127476 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:29.127483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:29.127786 (+ 303us) server_negotiation.cc:407] Connection header received
0926 20:16:29.127949 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:29.127953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:29.128042 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:29.128148 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:29.128679 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.129218 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.130047 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.130266 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.130346 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:29.133504 (+ 3158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:43889: BlockingRecv error: recv got EOF from 127.31.119.195:43889 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 20:16:29 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:29 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:29.725658 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:29.719013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:41659)
0926 20:16:29.719356 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:29.719371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:29.719475 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:29.719784 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:29.719786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:29.720018 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:29.720127 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:29.720133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.721000 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.721003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:29.721634 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:29.721644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.722189 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.722192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:29.722268 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:29.722918 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:29.722931 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:29.725468 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":123,"threads_started":1}
W20260926 20:16:29.725893 8948 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:29.719100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:41659 (local address 127.31.119.254:35793)
0926 20:16:29.719407 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:29.719411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:29.719429 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:16:29.719577 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:29.719580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:29.719628 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:29.719717 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:29.720272 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.720834 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.721776 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.722052 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.722152 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:29.725737 (+ 3585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:41659: BlockingRecv error: recv got EOF from 127.31.119.193:41659 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":115,"threads_started":1}
Sep 26 20:16:29 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:29 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:29.937827 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:29.931576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:45873)
0926 20:16:29.931908 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:29.931944 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:29.932077 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:29.932375 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:29.932379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:29.932615 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:29.932743 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:29.932757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.933805 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.933808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:29.934402 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:29.934412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.934946 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.934949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:29.935020 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:29.935509 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:29.935521 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:29.937703 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":82,"threads_started":1}
W20260926 20:16:29.938030 8948 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:29.931736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:45873 (local address 127.31.119.254:35793)
0926 20:16:29.931909 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:29.931914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:29.932043 (+ 129us) server_negotiation.cc:407] Connection header received
0926 20:16:29.932181 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:29.932184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:29.932244 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:29.932318 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:29.932890 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.933676 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.934547 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:29.934773 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:29.934856 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:29.937894 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:45873: BlockingRecv error: recv got EOF from 127.31.119.194:45873 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 20:16:30 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:30 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:30.141260 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:30.134591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:49841)
0926 20:16:30.134959 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:30.134973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:30.135243 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:30.135666 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:30.135669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:30.135879 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:30.135975 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:30.135982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.136800 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.136805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:30.137420 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:30.137428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.137914 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.137918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:30.137994 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:30.138594 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:30.138626 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:30.141111 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":125,"threads_started":1}
W20260926 20:16:30.141414 8948 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:30.134711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:49841 (local address 127.31.119.254:35793)
0926 20:16:30.134861 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:30.134866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:30.135106 (+ 240us) server_negotiation.cc:407] Connection header received
0926 20:16:30.135496 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:30.135499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:30.135550 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:30.135644 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:30.136102 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.136690 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.137552 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.137793 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.137908 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:30.141239 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:49841: BlockingRecv error: recv got EOF from 127.31.119.195:49841 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 20:16:30 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:30 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:30.734357 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:30.726573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:45337)
0926 20:16:30.726895 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:30.726914 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:30.727050 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:30.727399 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:30.727405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:30.727637 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:30.727769 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:30.727777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.729035 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.729041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:30.730169 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:30.730186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.731042 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.731047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:30.731161 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:30.731760 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:30.731780 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:30.734159 (+ 2379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
W20260926 20:16:30.734905 8952 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:30.726623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:45337 (local address 127.31.119.254:35793)
0926 20:16:30.726971 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:30.726976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:30.726995 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:16:30.727195 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:30.727199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:30.727264 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:30.727368 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:30.727908 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.728875 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.730337 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.730753 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.730878 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:30.734738 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:45337: BlockingRecv error: recv got EOF from 127.31.119.193:45337 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":57,"threads_started":1}
Sep 26 20:16:30 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:30 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:30.944593 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:30.938594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:41533)
0926 20:16:30.938999 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:30.939020 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:30.939123 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:30.939337 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:30.939339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:30.939523 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:30.939667 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:30.939672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.940579 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.940582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:30.941245 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:30.941255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.941694 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.941697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:30.941762 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:30.942366 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:30.942378 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:30.944420 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":118,"threads_started":1}
W20260926 20:16:30.944676 8952 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:30.938762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:41533 (local address 127.31.119.254:35793)
0926 20:16:30.938918 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:30.938924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:30.939102 (+ 178us) server_negotiation.cc:407] Connection header received
0926 20:16:30.939189 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:30.939191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:30.939235 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:30.939302 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:30.939772 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.940438 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.941356 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:30.941588 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:30.941672 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:30.944579 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:41533: BlockingRecv error: recv got EOF from 127.31.119.194:41533 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 20:16:31 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:31 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:31.149771 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:31.142199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:55129)
0926 20:16:31.142459 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:31.142475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:31.142620 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:31.142987 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:31.142991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:31.143218 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:31.143356 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:31.143368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.144590 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.144594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:31.145537 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:31.145551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.146259 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.146261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:31.146328 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:31.147204 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:31.147220 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:31.149554 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
W20260926 20:16:31.149863 8952 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:31.142242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:55129 (local address 127.31.119.254:35793)
0926 20:16:31.142387 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:31.142392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:31.142566 (+ 174us) server_negotiation.cc:407] Connection header received
0926 20:16:31.142746 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:31.142751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:31.142819 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:31.142909 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:31.143496 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.144448 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.145689 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.146025 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.146129 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:31.149735 (+ 3606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:55129: BlockingRecv error: recv got EOF from 127.31.119.195:55129 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 20:16:31 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:31 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:31.746834 8955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:31.735260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:51185)
0926 20:16:31.735560 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:31.735573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:31.735715 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:31.736129 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:31.736132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:31.736344 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:31.736442 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:31.736447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.737528 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.737531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:31.738105 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:31.738115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.738636 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.738639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:31.738705 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:31.744324 (+ 5619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:31.744340 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:31.746688 (+ 2348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
W20260926 20:16:31.747129 8956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:31.735330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:51185 (local address 127.31.119.254:35793)
0926 20:16:31.735623 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:31.735628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:31.735659 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:16:31.735869 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:31.735872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:31.735972 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:31.736083 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:31.736619 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.737308 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.738250 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.738518 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.738619 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:31.746942 (+ 8323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:51185: BlockingRecv error: recv got EOF from 127.31.119.193:51185 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":127,"threads_started":1}
Sep 26 20:16:31 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:31 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:31.951624 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:31.945508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:54259)
0926 20:16:31.945843 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:31.945860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:31.946044 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:31.946336 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:31.946338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:31.946576 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:31.946668 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:31.946675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.947451 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.947456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:31.948024 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:31.948032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.948509 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.948512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:31.948590 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:31.949151 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:31.949163 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:31.951431 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":88,"threads_started":1}
W20260926 20:16:31.951705 8956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:31.945665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:54259 (local address 127.31.119.254:35793)
0926 20:16:31.945833 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:31.945838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:31.945950 (+ 112us) server_negotiation.cc:407] Connection header received
0926 20:16:31.946162 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:31.946165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:31.946222 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:31.946312 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:31.946802 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.947336 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.948150 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:31.948383 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:31.948512 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:31.951587 (+ 3075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:54259: BlockingRecv error: recv got EOF from 127.31.119.194:54259 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 20:16:32 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:32 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:32.156243 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:32.150711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:45795)
0926 20:16:32.151023 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:32.151038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:32.151142 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:32.151406 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:32.151409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:32.151603 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:32.151704 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:32.151709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.152462 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.152501 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:32.153056 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:32.153065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.153547 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.153549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:32.153616 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:32.154209 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:32.154221 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:32.156120 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":73,"threads_started":1}
W20260926 20:16:32.156564 8956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:32.150800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:45795 (local address 127.31.119.254:35793)
0926 20:16:32.151002 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:32.151006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:32.151112 (+ 106us) server_negotiation.cc:407] Connection header received
0926 20:16:32.151217 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:32.151220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:32.151266 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:32.151337 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:32.151812 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.152343 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.153169 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.153426 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.153513 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:32.156360 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:45795: BlockingRecv error: recv got EOF from 127.31.119.195:45795 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 26 20:16:32 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:32 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:32.754055 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:32.747857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:46109)
0926 20:16:32.748259 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:32.748273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:32.748393 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:32.748751 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:32.748754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:32.748939 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:32.749065 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:32.749071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.750116 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.750119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:32.750816 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:32.750826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.751383 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.751386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:32.751454 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:32.751960 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:32.751977 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:32.753911 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":91,"threads_started":1}
W20260926 20:16:32.754269 8960 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:32.748050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:46109 (local address 127.31.119.254:35793)
0926 20:16:32.748297 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:32.748301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:32.748358 (+ 57us) server_negotiation.cc:407] Connection header received
0926 20:16:32.748535 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:32.748538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:32.748590 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:32.748683 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:32.749204 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.749978 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.750937 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.751279 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.751364 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:32.754090 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:46109: BlockingRecv error: recv got EOF from 127.31.119.193:46109 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":60,"threads_started":1}
Sep 26 20:16:32 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:32 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:32.958619 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:32.952532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:33395)
0926 20:16:32.952920 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:32.952933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:32.953107 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:32.953413 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:32.953416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:32.953647 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:32.953740 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:32.953746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.954551 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.954556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:32.955118 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:32.955125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.955594 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.955598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:32.955674 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:32.956203 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:32.956216 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:32.958373 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":89,"threads_started":1}
W20260926 20:16:32.958797 8960 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:32.952657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:33395 (local address 127.31.119.254:35793)
0926 20:16:32.952845 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:32.952850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:32.953042 (+ 192us) server_negotiation.cc:407] Connection header received
0926 20:16:32.953213 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:32.953217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:32.953273 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:32.953389 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:32.953863 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.954399 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.955240 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:32.955468 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:32.955593 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:32.958508 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:33395: BlockingRecv error: recv got EOF from 127.31.119.194:33395 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 20:16:33 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:33 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:33.162878 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:33.157145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:56995)
0926 20:16:33.157502 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:33.157515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:33.157621 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:33.157918 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:33.157922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:33.158121 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:33.158238 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:33.158244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.159059 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.159062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:33.159622 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:33.159631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.160071 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.160074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:33.160134 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:33.160786 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:33.160799 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:33.162704 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
W20260926 20:16:33.162964 8960 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:33.157234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:56995 (local address 127.31.119.254:35793)
0926 20:16:33.157387 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:33.157392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:33.157592 (+ 200us) server_negotiation.cc:407] Connection header received
0926 20:16:33.157703 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:33.157705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:33.157753 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:33.157824 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:33.158359 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.158947 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.159733 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.159953 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.160044 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:33.162833 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:56995: BlockingRecv error: recv got EOF from 127.31.119.195:56995 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 20:16:33 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:33 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:33.761442 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:33.755048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:39747)
0926 20:16:33.755350 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:33.755373 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:33.755501 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:33.755904 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:33.755908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:33.756151 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:33.756305 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:33.756317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.757189 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.757193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:33.757843 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:33.757900 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.758466 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.758470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:33.758571 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:33.759212 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:33.759225 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:33.761238 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":109,"threads_started":1}
W20260926 20:16:33.761577 8964 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:33.755253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:39747 (local address 127.31.119.254:35793)
0926 20:16:33.755555 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:33.755558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:33.755593 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:16:33.755634 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:33.755637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:33.755690 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:33.755792 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:33.756448 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.757057 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.758042 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.758324 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.758463 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:33.761437 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:39747: BlockingRecv error: recv got EOF from 127.31.119.193:39747 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
Sep 26 20:16:33 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:33 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:33.965708 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:33.959608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:57851)
0926 20:16:33.959895 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:33.959907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:33.960022 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:33.960280 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:33.960282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:33.960459 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:33.960557 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:33.960562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.961613 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.961619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:33.962348 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:33.962360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.962985 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.962988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:33.963054 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:33.963677 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:33.963689 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:33.965588 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":117,"threads_started":1}
W20260926 20:16:33.965896 8964 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:33.959610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:57851 (local address 127.31.119.254:35793)
0926 20:16:33.959786 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:33.959790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:33.959984 (+ 194us) server_negotiation.cc:407] Connection header received
0926 20:16:33.960118 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:33.960120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:33.960167 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:33.960245 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:33.960664 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.961485 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.962559 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:33.962865 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:33.962963 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:33.965735 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:57851: BlockingRecv error: recv got EOF from 127.31.119.194:57851 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 20:16:34 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:34 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:34.170560 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:34.164025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:59773)
0926 20:16:34.164298 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:34.164321 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:34.164429 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:34.164753 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:34.164757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:34.164962 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:34.165071 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:34.165077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.166059 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.166062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:34.166788 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:34.166804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.167370 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.167373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:34.167474 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:34.168098 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:34.168112 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:34.170434 (+ 2322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":103,"threads_started":1}
W20260926 20:16:34.170866 8964 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:34.164185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:59773 (local address 127.31.119.254:35793)
0926 20:16:34.164326 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:34.164329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:34.164397 (+ 68us) server_negotiation.cc:407] Connection header received
0926 20:16:34.164512 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:34.164515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:34.164563 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:34.164650 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:34.165206 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.165925 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.166925 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.167244 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.167330 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:34.170738 (+ 3408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:59773: BlockingRecv error: recv got EOF from 127.31.119.195:59773 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 20:16:34 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:34 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:34.768327 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:34.762305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:45527)
0926 20:16:34.762579 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:34.762596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:34.762702 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:34.762928 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:34.762930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:34.763137 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:34.763236 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:34.763241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.764067 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.764070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:34.764661 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:34.764670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.765214 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.765217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:34.765350 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:34.765935 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:34.765947 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:34.768134 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
W20260926 20:16:34.768440 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:34.762480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:45527 (local address 127.31.119.254:35793)
0926 20:16:34.762703 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:34.762707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:34.762720 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:16:34.762761 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:34.762763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:34.762810 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:34.762895 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:34.763366 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.763939 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.764786 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.765038 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.765124 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:34.768294 (+ 3170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:45527: BlockingRecv error: recv got EOF from 127.31.119.193:45527 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
Sep 26 20:16:34 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:34 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:34.972518 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:34.966488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:38041)
0926 20:16:34.966768 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:34.966781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:34.966919 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:34.967234 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:34.967236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:34.967421 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:34.967565 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:34.967577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.968531 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.968534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:34.969096 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:34.969105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.969681 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.969684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:34.969748 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:34.970273 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:34.970285 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:34.972340 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
W20260926 20:16:34.972754 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:34.966624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:38041 (local address 127.31.119.254:35793)
0926 20:16:34.966810 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:34.966813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:34.966857 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:16:34.967011 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:34.967014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:34.967057 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:34.967142 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:34.967748 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.968341 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.969240 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:34.969534 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:34.969637 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:34.972596 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:38041: BlockingRecv error: recv got EOF from 127.31.119.194:38041 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 20:16:35 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:35 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:35.177521 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:35.171549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:44249)
0926 20:16:35.171859 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:35.171873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:35.171973 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:35.172302 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:35.172304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:35.172496 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:35.172589 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:35.172594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.173477 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.173480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:35.174041 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:35.174050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.174517 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.174519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:35.174580 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:35.175153 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:35.175165 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:35.177394 (+ 2229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":115,"threads_started":1}
W20260926 20:16:35.177675 8968 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:35.171615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:44249 (local address 127.31.119.254:35793)
0926 20:16:35.171761 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:35.171766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:35.171969 (+ 203us) server_negotiation.cc:407] Connection header received
0926 20:16:35.172112 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:35.172114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:35.172176 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:35.172277 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:35.172700 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.173236 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.174164 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.174410 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.174498 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:35.177549 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:44249: BlockingRecv error: recv got EOF from 127.31.119.195:44249 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 20:16:35 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:35 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:35.775319 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:35.769410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:37547)
0926 20:16:35.769726 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:35.769787 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:35.769909 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:35.770247 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:35.770249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:35.770497 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:35.770598 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:35.770604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.771400 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.771405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:35.771974 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:35.771982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.772479 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.772483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:35.772561 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:35.773144 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:35.773157 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:35.775199 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":130,"threads_started":1}
W20260926 20:16:35.775609 8972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:35.769573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:37547 (local address 127.31.119.254:35793)
0926 20:16:35.769882 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:35.769889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:35.769909 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:16:35.770031 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:35.770034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:35.770089 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:35.770205 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:35.770734 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.771273 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.772103 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.772362 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.772478 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:35.775444 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:37547: BlockingRecv error: recv got EOF from 127.31.119.193:37547 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":103,"threads_started":1}
Sep 26 20:16:35 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:35 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:35.979458 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:35.973549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:57769)
0926 20:16:35.973821 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:35.973844 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:35.973936 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:35.974203 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:35.974205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:35.974404 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:35.974529 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:35.974535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.975420 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.975423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:35.976159 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:35.976169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.976645 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.976648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:35.976711 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:35.977207 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:35.977219 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:35.979246 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
W20260926 20:16:35.979499 8972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:35.973736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:57769 (local address 127.31.119.254:35793)
0926 20:16:35.973888 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:35.973892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:35.973908 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:16:35.974030 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:35.974033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:35.974079 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:35.974166 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:35.974662 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.975297 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.976281 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:35.976537 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:35.976626 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:35.979391 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:57769: BlockingRecv error: recv got EOF from 127.31.119.194:57769 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 20:16:36 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:36 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:36.184589 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:36.178496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:33747)
0926 20:16:36.178861 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:36.178874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:36.178972 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:36.179276 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:36.179278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:36.179536 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:36.179641 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:36.179648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.180443 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.180446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:36.181268 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:36.181305 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.181803 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.181807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:36.181904 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:36.182490 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:36.182502 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:36.184473 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":194,"threads_started":1}
W20260926 20:16:36.184800 8972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:36.178671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:33747 (local address 127.31.119.254:35793)
0926 20:16:36.178815 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:36.178819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:36.178940 (+ 121us) server_negotiation.cc:407] Connection header received
0926 20:16:36.179106 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:36.179109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:36.179160 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:36.179245 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:36.179756 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.180318 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.181406 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.181694 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.181800 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:36.184652 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:33747: BlockingRecv error: recv got EOF from 127.31.119.195:33747 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 20:16:36 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:36 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:36.786151 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:36.776245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:37721)
0926 20:16:36.776556 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:36.776569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:36.776670 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:36.776969 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:36.776972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:36.777151 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:36.777266 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:36.777271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.778087 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.778089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:36.778703 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:36.778712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.779172 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.779175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:36.779243 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:36.783974 (+ 4731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:36.783987 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:36.786033 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":85,"threads_started":1}
W20260926 20:16:36.786388 8976 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:36.776381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:37721 (local address 127.31.119.254:35793)
0926 20:16:36.776620 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:36.776624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:36.776639 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:36.776768 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:36.776770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:36.776815 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:36.776896 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:36.777417 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.777980 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.778806 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.779041 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.779120 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:36.786228 (+ 7108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:37721: BlockingRecv error: recv got EOF from 127.31.119.193:37721 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":59,"threads_started":1}
Sep 26 20:16:36 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:36 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:36.986105 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:36.980445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:41931)
0926 20:16:36.980748 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:36.980761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:36.980872 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:36.981189 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:36.981192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:36.981437 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:36.981577 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:36.981586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.982392 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.982395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:36.982963 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:36.982971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.983415 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.983418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:36.983480 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:36.984023 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:36.984034 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:36.985910 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":114,"threads_started":1}
W20260926 20:16:36.986205 8976 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:36.980601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:41931 (local address 127.31.119.254:35793)
0926 20:16:36.980760 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:36.980764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:36.980823 (+ 59us) server_negotiation.cc:407] Connection header received
0926 20:16:36.980970 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:36.980972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:36.981043 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:36.981117 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:36.981726 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.982281 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.983078 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:36.983313 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:36.983396 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:36.986077 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:41931: BlockingRecv error: recv got EOF from 127.31.119.194:41931 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 20:16:37 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:37 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:37.191512 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:37.185453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:47653)
0926 20:16:37.185756 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:37.185770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:37.185872 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:37.186150 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:37.186152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:37.186332 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:37.186465 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:37.186471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.187185 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.187188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:37.187781 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:37.187790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.188231 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.188234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:37.188296 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:37.188962 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:37.188976 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:37.191299 (+ 2323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":121,"threads_started":1}
W20260926 20:16:37.191701 8976 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:37.185606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:47653 (local address 127.31.119.254:35793)
0926 20:16:37.185744 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:37.185748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:37.185848 (+ 100us) server_negotiation.cc:407] Connection header received
0926 20:16:37.185962 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:37.185965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:37.186014 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:37.186093 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:37.186568 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.187093 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.187900 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.188127 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.188209 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:37.191525 (+ 3316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:47653: BlockingRecv error: recv got EOF from 127.31.119.195:47653 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 20:16:37 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:37 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:37.793252 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:37.786976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:50359)
0926 20:16:37.787304 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:37.787317 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:37.787418 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:37.787682 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:37.787684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:37.787883 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:37.787980 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:37.787986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.788784 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.788787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:37.789451 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:37.789461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.789913 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.789916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:37.789979 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:37.790639 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:37.790654 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:37.793080 (+ 2426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
W20260926 20:16:37.793376 8980 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:37.787127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:50359 (local address 127.31.119.254:35793)
0926 20:16:37.787364 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:37.787368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:37.787383 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:37.787511 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:37.787514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:37.787559 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:37.787640 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:37.788093 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.788673 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.789563 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.789790 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.789875 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:37.793214 (+ 3339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:50359: BlockingRecv error: recv got EOF from 127.31.119.193:50359 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":55,"threads_started":1}
Sep 26 20:16:37 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:37 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:37.992744 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:37.986883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:44313)
0926 20:16:37.987222 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:37.987235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:37.987431 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:37.987715 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:37.987718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:37.987932 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:37.988024 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:37.988030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.988893 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.988897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:37.989528 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:37.989537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.990010 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.990014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:37.990096 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:37.990607 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:37.990619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:37.992555 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
W20260926 20:16:37.992808 8980 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:37.986993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:44313 (local address 127.31.119.254:35793)
0926 20:16:37.987167 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:37.987172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:37.987309 (+ 137us) server_negotiation.cc:407] Connection header received
0926 20:16:37.987546 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:37.987549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:37.987605 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:37.987691 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:37.988142 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.988772 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.989653 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:37.989897 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:37.990009 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:37.992699 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:44313: BlockingRecv error: recv got EOF from 127.31.119.194:44313 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 20:16:38 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:38 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:38.198872 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:38.192698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:35969)
0926 20:16:38.192957 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:38.192971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:38.193119 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:38.193492 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:38.193494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:38.193714 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:38.193807 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:38.193813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.194623 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.194628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:38.195194 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:38.195203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.195670 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.195673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:38.195766 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:38.196372 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:38.196384 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:38.198746 (+ 2362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
W20260926 20:16:38.199131 8980 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:38.192743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:35969 (local address 127.31.119.254:35793)
0926 20:16:38.192924 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:38.192928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:38.193056 (+ 128us) server_negotiation.cc:407] Connection header received
0926 20:16:38.193220 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:38.193224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:38.193306 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:38.193475 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:38.193927 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.194502 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.195312 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.195553 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.195664 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:38.198989 (+ 3325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:35969: BlockingRecv error: recv got EOF from 127.31.119.195:35969 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 20:16:38 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:38 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:38.800374 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:38.794187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:47587)
0926 20:16:38.794515 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:38.794529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:38.794635 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:38.794897 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:38.794900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:38.795087 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:38.795183 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:38.795189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.796079 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.796082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:38.796737 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:38.796748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.797332 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.797337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:38.797459 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:38.798104 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:38.798117 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:38.800224 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
W20260926 20:16:38.800591 8984 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:38.794320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:47587 (local address 127.31.119.254:35793)
0926 20:16:38.794652 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:38.794656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:38.794672 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:16:38.794716 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:38.794718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:38.794764 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:38.794866 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:38.795321 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.795951 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.796931 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.797178 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.797269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:38.800435 (+ 3166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:47587: BlockingRecv error: recv got EOF from 127.31.119.193:47587 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":96,"threads_started":1}
Sep 26 20:16:38 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:38 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:38.999631 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:38.993682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:58675)
0926 20:16:38.993978 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:38.993992 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:38.994089 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:38.994333 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:38.994336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:38.994521 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:38.994618 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:38.994623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.995448 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.995450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:38.996205 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:38.996216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.996723 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.996747 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:38.996817 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:38.997429 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:38.997441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:38.999506 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":113,"threads_started":1}
W20260926 20:16:38.999788 8984 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:38.993852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:58675 (local address 127.31.119.254:35793)
0926 20:16:38.993998 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:38.994002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:38.994044 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:16:38.994172 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:38.994174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:38.994217 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:38.994288 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:38.994789 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.995328 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.996378 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:38.996611 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:38.996695 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:38.999690 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:58675: BlockingRecv error: recv got EOF from 127.31.119.194:58675 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 20:16:39 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:39 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:39.205755 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:39.199841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:50679)
0926 20:16:39.200128 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:39.200147 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:39.200241 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:39.200561 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:39.200563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:39.200739 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:39.200834 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:39.200839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:39.201646 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:39.201648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:39.202208 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:39.202216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:39.202727 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:39.202729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:39.202790 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:39.203483 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:39.203495 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:39.205607 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":140,"threads_started":1}
W20260926 20:16:39.205999 8984 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:39.199980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:50679 (local address 127.31.119.254:35793)
0926 20:16:39.200133 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:39.200137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:39.200198 (+ 61us) server_negotiation.cc:407] Connection header received
0926 20:16:39.200387 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:39.200390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:39.200440 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:39.200520 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:39.200961 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:39.201538 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:39.202381 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:39.202622 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:39.202708 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:39.205852 (+ 3144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:50679: BlockingRecv error: recv got EOF from 127.31.119.195:50679 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 20:16:39 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:39 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:39.807194 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:39.801234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:39863)
0926 20:16:39.801593 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:39.801607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:39.801703 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:39.802003 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:39.802006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:39.802186 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:39.802354 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:39.802362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:39.803168 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:39.803171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:39.803828 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:39.803840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:39.804281 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:39.804283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:39.804402 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:39.805011 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:39.805023 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:39.807012 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":167,"threads_started":1}
W20260926 20:16:39.807269 8988 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:39.801388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:39863 (local address 127.31.119.254:35793)
0926 20:16:39.801682 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:39.801686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:39.801701 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:16:39.801793 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:39.801796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:39.801841 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:39.801930 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:39.802493 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:39.803041 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:39.803946 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:39.804178 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:39.804256 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:39.807142 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:39863: BlockingRecv error: recv got EOF from 127.31.119.193:39863 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":84,"threads_started":1}
Sep 26 20:16:40 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:40 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:40.006747 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:40.000810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:55153)
0926 20:16:40.001102 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:40.001116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:40.001222 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:40.001720 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:40.001722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:40.001909 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:40.002007 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:40.002013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.002853 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.002855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:40.003453 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:40.003462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.003962 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.003965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:40.004044 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:40.004654 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:40.004667 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:40.006614 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":104,"threads_started":1}
W20260926 20:16:40.006953 8988 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:40.000849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:55153 (local address 127.31.119.254:35793)
0926 20:16:40.001007 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:40.001012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:40.001198 (+ 186us) server_negotiation.cc:407] Connection header received
0926 20:16:40.001340 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:40.001342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:40.001389 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:40.001462 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:40.002145 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.002737 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.003591 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.003813 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.003906 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:40.006787 (+ 2881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:55153: BlockingRecv error: recv got EOF from 127.31.119.194:55153 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 20:16:40 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:40 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.195@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:40.212364 8990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:40.206662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.195:45377)
0926 20:16:40.206930 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:40.206942 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:40.207040 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:40.207279 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:40.207281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:40.207495 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:40.207590 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:40.207595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.208329 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.208331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:40.208933 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:40.208941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.209442 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.209445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:40.209519 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:40.210046 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:40.210058 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:40.212163 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":99,"threads_started":1}
W20260926 20:16:40.212513 8988 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:40.206797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:45377 (local address 127.31.119.254:35793)
0926 20:16:40.206937 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:40.206940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:40.207012 (+ 72us) server_negotiation.cc:407] Connection header received
0926 20:16:40.207118 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:40.207121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:40.207167 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:40.207242 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:40.207694 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.208236 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.209054 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.209318 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.209417 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:40.212394 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.195:45377: BlockingRecv error: recv got EOF from 127.31.119.195:45377 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 20:16:40 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:40 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.193@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:40.814002 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:40.808194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.193:42085)
0926 20:16:40.808507 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:40.808519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:40.808612 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:40.808844 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:40.808846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:40.809062 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:40.809157 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:40.809163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.809958 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.809961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:40.810561 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:40.810572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.811021 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.811023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:40.811087 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:40.811623 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:40.811635 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:40.813870 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":157,"threads_started":1}
W20260926 20:16:40.814246 8992 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:40.808253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:42085 (local address 127.31.119.254:35793)
0926 20:16:40.808517 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:40.808524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:40.808574 (+ 50us) server_negotiation.cc:407] Connection header received
0926 20:16:40.808689 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:40.808691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:40.808734 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:40.808843 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:40.809312 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.809847 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.810675 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:40.810920 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:40.811004 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:40.814058 (+ 3054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.193:42085: BlockingRecv error: recv got EOF from 127.31.119.193:42085 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":58,"threads_started":1}
Sep 26 20:16:41 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:16:41 dist-test-slave-vnbh krb5kdc[7933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.119.194@KRBTEST.COM for oryx/127.31.119.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:16:41.014597 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:41.008015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:35793 (local address 127.31.119.194:50121)
0926 20:16:41.008422 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:41.008440 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:41.008575 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:41.008884 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:41.008886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:41.009084 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:41.009182 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:41.009187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.010204 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.010209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:41.011000 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:41.011011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.011510 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.011514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:41.011603 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:41.012148 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:41.012166 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:41.014402 (+ 2236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.119.254:35793: Server oryx/127.31.119.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":77,"threads_started":1}
W20260926 20:16:41.014683 8992 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:41.008184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:50121 (local address 127.31.119.254:35793)
0926 20:16:41.008431 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:41.008435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:41.008560 (+ 125us) server_negotiation.cc:407] Connection header received
0926 20:16:41.008669 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:41.008672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:41.008721 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:41.008803 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:41.009364 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.010071 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.011123 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.011383 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.011501 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:41.014557 (+ 3056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.119.194:50121: BlockingRecv error: recv got EOF from 127.31.119.194:50121 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 20:16:41.093763 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 8517
I20260926 20:16:41.099620 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 8669
I20260926 20:16:41.105202 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 8801
I20260926 20:16:41.110674 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 8447
2026-09-26T20:16:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18166 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T20:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:41Z Disabled control of system clock
I20260926 20:16:41.141853 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:41541
--webserver_interface=127.31.119.254
--webserver_port=0
--builtin_ntp_servers=127.31.119.212:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:41541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:16:41.271198 8997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:41.271492 8997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:41.271528 8997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:41.275619 8997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:41.275693 8997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:41.275709 8997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:41.275727 8997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:41.275743 8997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:41.280716 8997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:41541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:41541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:41.281863 8997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:41.283298 8997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:41.292886 9002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:41.292951 9005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:41.293009 8997 server_base.cc:1034] running on GCE node
W20260926 20:16:41.293133 9003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:41.293810 8997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:41.294965 8997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:41.296137 8997 hybrid_clock.cc:648] HybridClock initialized: now 1790453801296120 us; error 34 us; skew 500 ppm
I20260926 20:16:41.298168 8997 webserver.cc:460] Webserver started at http://127.31.119.254:45817/ using document root <none> and password file <none>
I20260926 20:16:41.298739 8997 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:41.298802 8997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:41.298977 8997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:41.301079 8997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "c54c23a4603c44e1a7c53881181e5643"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "594230ba4cd32d1ebb770087736bb1fb"
server_key_iv: "5bfb8d22a2edd633944cfb708d3814c6"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.301693 8997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "c54c23a4603c44e1a7c53881181e5643"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "594230ba4cd32d1ebb770087736bb1fb"
server_key_iv: "5bfb8d22a2edd633944cfb708d3814c6"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.305778 8997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:41.308688 9011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:41.310154 8997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:41.310284 8997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "c54c23a4603c44e1a7c53881181e5643"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "594230ba4cd32d1ebb770087736bb1fb"
server_key_iv: "5bfb8d22a2edd633944cfb708d3814c6"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.310389 8997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:41.322518 8997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:41.323222 8997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:41.323390 8997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:41.330693 8997 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:41541
I20260926 20:16:41.330718 9063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:41541 every 8 connection(s)
I20260926 20:16:41.331705 8997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:41.335129 9064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:41.338172 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 8997
I20260926 20:16:41.338310 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:41.338603 32223 external_mini_cluster.cc:1428] Setting key 73681a9066f90734915d2aad59419bd1
I20260926 20:16:41.341481 9064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643: Bootstrap starting.
I20260926 20:16:41.343778 9064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:41.344748 9064 log.cc:824] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:41.347116 9064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643: No bootstrap required, opened a new log
I20260926 20:16:41.348701 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:41.340117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.31.119.254:41541)
0926 20:16:41.340793 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:41.340809 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:41.340852 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:16:41.341561 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:41.341581 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:41.342063 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:41.342416 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:41.342865 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.343834 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.345099 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.345796 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.345916 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:41.346844 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:41.346848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:41.346862 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:41.347285 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:41.347353 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:41.347374 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:41.347482 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:41.347657 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 20:16:41.347953 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":138,"threads_started":1}
I20260926 20:16:41.350801 9064 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c23a4603c44e1a7c53881181e5643" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41541 } }
I20260926 20:16:41.351089 9064 raft_consensus.cc:374] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:41.351146 9064 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54c23a4603c44e1a7c53881181e5643, State: Initialized, Role: FOLLOWER
I20260926 20:16:41.351547 9064 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c23a4603c44e1a7c53881181e5643" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41541 } }
I20260926 20:16:41.351665 9064 raft_consensus.cc:388] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:41.351709 9064 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:41.351779 9064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:41.352660 9064 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c23a4603c44e1a7c53881181e5643" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41541 } }
I20260926 20:16:41.352954 9064 leader_election.cc:302] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c54c23a4603c44e1a7c53881181e5643; no voters:
I20260926 20:16:41.353229 9064 leader_election.cc:288] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:41.353387 9069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:41.353632 9069 raft_consensus.cc:686] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [term 1 LEADER]: Becoming Leader. State: Replica: c54c23a4603c44e1a7c53881181e5643, State: Running, Role: LEADER
I20260926 20:16:41.354050 9069 consensus_queue.cc:237] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c23a4603c44e1a7c53881181e5643" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41541 } }
I20260926 20:16:41.354557 9064 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:41.355756 9071 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54c23a4603c44e1a7c53881181e5643. Latest consensus state: current_term: 1 leader_uuid: "c54c23a4603c44e1a7c53881181e5643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c23a4603c44e1a7c53881181e5643" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41541 } } }
I20260926 20:16:41.355891 9071 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:41.356263 9078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:41.356242 9070 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c54c23a4603c44e1a7c53881181e5643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54c23a4603c44e1a7c53881181e5643" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41541 } } }
I20260926 20:16:41.356375 9070 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:41.359390 9078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:41.365396 9078 catalog_manager.cc:1347] Generated new cluster ID: 739efcc18be84663b4e01e9c7a0bdc9f
I20260926 20:16:41.365479 9078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:41.377235 9078 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:41.378567 9078 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:41.386324 9078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c54c23a4603c44e1a7c53881181e5643: Generated new TSK 0
I20260926 20:16:41.387228 9078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:16:41.391907 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41541
--builtin_ntp_servers=127.31.119.212:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 20:16:41.511143 9088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:41.511453 9088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:41.511492 9088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:41.515975 9088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:41.516049 9088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:41.516108 9088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:41.521211 9088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:41.522441 9088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:41.523875 9088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:41.533702 9093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:41.533720 9094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:41.533717 9096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:41.533787 9088 server_base.cc:1034] running on GCE node
I20260926 20:16:41.534368 9088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:41.535064 9088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:41.536263 9088 hybrid_clock.cc:648] HybridClock initialized: now 1790453801536244 us; error 31 us; skew 500 ppm
I20260926 20:16:41.538231 9088 webserver.cc:460] Webserver started at http://127.31.119.193:36957/ using document root <none> and password file <none>
I20260926 20:16:41.538843 9088 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:41.538938 9088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:41.539172 9088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:41.541276 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "1e896606cae8450fa46526a4ea4eb086"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "3307ae49c3a7e77d15bf2477c742876a"
server_key_iv: "828373d798c30bdaed92c9eb24f4e3e7"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.541937 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "1e896606cae8450fa46526a4ea4eb086"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "3307ae49c3a7e77d15bf2477c742876a"
server_key_iv: "828373d798c30bdaed92c9eb24f4e3e7"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.545872 9088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:16:41.548719 9102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:41.550277 9088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:41.550448 9088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "1e896606cae8450fa46526a4ea4eb086"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "3307ae49c3a7e77d15bf2477c742876a"
server_key_iv: "828373d798c30bdaed92c9eb24f4e3e7"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.550587 9088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:41.599634 9088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:41.600425 9088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:41.600647 9088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:41.601310 9088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:41.602361 9088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:41.602434 9088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:41.602505 9088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:41.602547 9088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:41.611868 9088 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:40075
I20260926 20:16:41.611892 9215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:40075 every 8 connection(s)
I20260926 20:16:41.612880 9088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:41.620066 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 9088
I20260926 20:16:41.620205 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:16:41.620637 32223 external_mini_cluster.cc:1428] Setting key 192d8463e98dcd573f950e5ded68ad40
I20260926 20:16:41.623143 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41541
--builtin_ntp_servers=127.31.119.212:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:16:41.624368 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:41.615053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:54203 (local address 127.31.119.254:41541)
0926 20:16:41.615193 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:41.615197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:41.616429 (+ 1232us) server_negotiation.cc:407] Connection header received
0926 20:16:41.618171 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:41.618175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:41.618236 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:41.618322 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:41.619355 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.619925 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.621661 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.621973 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.622083 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:41.623864 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:41.623867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:41.623869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:41.623882 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:41.623904 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:41.623912 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:41.623980 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:41.624188 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 20:16:41.624249 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:16:41.625576 9218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:41.615460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41541 (local address 127.31.119.193:54203)
0926 20:16:41.616155 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:41.616199 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:41.617900 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:41.618553 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:41.618560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:41.618654 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:41.619201 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:41.619215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.620083 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.620090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:41.621530 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:41.621541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.622284 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.622287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:41.622646 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:41.623490 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:41.623494 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:41.623559 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:41.623664 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:41.624012 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:41.624025 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:41.624035 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:16:41.624180 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 20:16:41.624546 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260926 20:16:41.626905 9216 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41541
I20260926 20:16:41.627187 9216 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:41.627910 9216 heartbeater.cc:511] Master 127.31.119.254:41541 requested a full tablet report, sending...
I20260926 20:16:41.629742 9028 ts_manager.cc:194] Registered new tserver with Master: 1e896606cae8450fa46526a4ea4eb086 (127.31.119.193:40075)
I20260926 20:16:41.631011 9028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.193:54203
W20260926 20:16:41.739151 9219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:41.739444 9219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:41.739483 9219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:41.744590 9219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:41.744681 9219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:41.744745 9219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:41.750174 9219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:41.751420 9219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:41.752910 9219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:41.763618 9227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:41.763774 9219 server_base.cc:1034] running on GCE node
W20260926 20:16:41.763635 9225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:41.763664 9224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:41.764524 9219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:41.765342 9219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:41.766572 9219 hybrid_clock.cc:648] HybridClock initialized: now 1790453801766556 us; error 27 us; skew 500 ppm
I20260926 20:16:41.768800 9219 webserver.cc:460] Webserver started at http://127.31.119.194:38087/ using document root <none> and password file <none>
I20260926 20:16:41.769524 9219 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:41.769630 9219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:41.769932 9219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:41.772230 9219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "455ba874c2e54bb78ca7e860b9cb1f9d"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "3af96c47e3f64b696318ced43b9756b8"
server_key_iv: "2d6fd98c48b3587f4b8ce265b21c2201"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.772856 9219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "455ba874c2e54bb78ca7e860b9cb1f9d"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "3af96c47e3f64b696318ced43b9756b8"
server_key_iv: "2d6fd98c48b3587f4b8ce265b21c2201"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.776906 9219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:16:41.779686 9233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:41.781232 9219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:16:41.781491 9219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "455ba874c2e54bb78ca7e860b9cb1f9d"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "3af96c47e3f64b696318ced43b9756b8"
server_key_iv: "2d6fd98c48b3587f4b8ce265b21c2201"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.781630 9219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:41.807242 9219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:41.808043 9219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:41.808256 9219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:41.808876 9219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:41.809888 9219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:41.809962 9219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:41.810031 9219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:41.810082 9219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:41.819346 9219 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:41549
I20260926 20:16:41.819368 9346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:41549 every 8 connection(s)
I20260926 20:16:41.820348 9219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:41.820951 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 9219
I20260926 20:16:41.821069 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:41.821446 32223 external_mini_cluster.cc:1428] Setting key 10d3466dc9dc61434932e4fe11bd7c92
I20260926 20:16:41.823659 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41541
--builtin_ntp_servers=127.31.119.212:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:16:41.831183 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:41.822558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:37427 (local address 127.31.119.254:41541)
0926 20:16:41.822703 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:41.822709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:41.824177 (+ 1468us) server_negotiation.cc:407] Connection header received
0926 20:16:41.826207 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:41.826210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:41.826259 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:41.826463 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:41.827422 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.827954 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.829122 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.829401 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.829511 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:41.830661 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:41.830664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:41.830667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:41.830681 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:41.830709 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:41.830717 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:41.830791 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:41.830970 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:16:41.831032 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:16:41.832054 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:41.823067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41541 (local address 127.31.119.194:37427)
0926 20:16:41.823994 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:41.824065 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:41.825986 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:41.826682 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:41.826689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:41.826778 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:41.827282 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:41.827295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.828081 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.828087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:41.828995 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:41.829005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:41.829563 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:41.829568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:41.829734 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:41.830374 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:41.830375 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:41.830417 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:41.830489 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:41.830804 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:41.830814 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:41.830819 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:16:41.830950 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 20:16:41.831235 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":131,"threads_started":1}
I20260926 20:16:41.833318 9347 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41541
I20260926 20:16:41.833592 9347 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:41.834143 9347 heartbeater.cc:511] Master 127.31.119.254:41541 requested a full tablet report, sending...
I20260926 20:16:41.835281 9028 ts_manager.cc:194] Registered new tserver with Master: 455ba874c2e54bb78ca7e860b9cb1f9d (127.31.119.194:41549)
I20260926 20:16:41.836055 9028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.194:37427
W20260926 20:16:41.942415 9350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:41.942691 9350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:41.942729 9350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:41.946729 9350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:41.946785 9350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:41.946838 9350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:41.951731 9350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:41.952881 9350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:41.954370 9350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:41.964879 9358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:41.964948 9356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:41.964879 9355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:41.965436 9350 server_base.cc:1034] running on GCE node
I20260926 20:16:41.965878 9350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:41.966567 9350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:41.967792 9350 hybrid_clock.cc:648] HybridClock initialized: now 1790453801967755 us; error 60 us; skew 500 ppm
I20260926 20:16:41.970043 9350 webserver.cc:460] Webserver started at http://127.31.119.195:45767/ using document root <none> and password file <none>
I20260926 20:16:41.970669 9350 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:41.970732 9350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:41.970959 9350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:41.973066 9350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "6e4f50199eae464e8eb9bb34b5e8e035"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "8a4d0cbbdf02226f6da15f10c0473e0a"
server_key_iv: "29b8a14bcc57c106eddc7486cda9fad4"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.973691 9350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "6e4f50199eae464e8eb9bb34b5e8e035"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "8a4d0cbbdf02226f6da15f10c0473e0a"
server_key_iv: "29b8a14bcc57c106eddc7486cda9fad4"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.977643 9350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:41.980450 9364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:41.982008 9350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:41.982154 9350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "6e4f50199eae464e8eb9bb34b5e8e035"
format_stamp: "Formatted at 2026-09-26 20:16:41 on dist-test-slave-vnbh"
server_key: "8a4d0cbbdf02226f6da15f10c0473e0a"
server_key_iv: "29b8a14bcc57c106eddc7486cda9fad4"
server_key_version: "encryptionkey@0"
I20260926 20:16:41.982290 9350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:42.005388 9350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:42.006217 9350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:42.006444 9350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:42.007097 9350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:42.008095 9350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:42.008165 9350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.008241 9350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:42.008294 9350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.018118 9350 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:39349
I20260926 20:16:42.018141 9477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:39349 every 8 connection(s)
I20260926 20:16:42.019230 9350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:42.020857 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 9350
I20260926 20:16:42.020975 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:42.021284 32223 external_mini_cluster.cc:1428] Setting key a0672691f5280845478b753aea6d1420
I20260926 20:16:42.029503 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:42.021403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:50765 (local address 127.31.119.254:41541)
0926 20:16:42.021553 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:42.021558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:42.023104 (+ 1546us) server_negotiation.cc:407] Connection header received
0926 20:16:42.024613 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:42.024617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:42.024672 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:42.024751 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:42.025744 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:42.026277 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:42.027454 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:42.027682 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:42.027796 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:42.028928 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:42.028931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:42.028934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:42.028948 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:42.028975 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:42.028983 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:42.029090 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:42.029254 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:16:42.029341 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:16:42.030411 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:42.022041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41541 (local address 127.31.119.195:50765)
0926 20:16:42.022939 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:42.023007 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:42.024406 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:42.024931 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:42.024939 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:42.025035 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:42.025604 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:42.025617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:42.026416 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:42.026420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:42.027321 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:42.027330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:42.027837 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:42.027841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:42.028010 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:42.028665 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:42.028667 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:42.028708 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:42.028772 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:42.029090 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:42.029101 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:42.029109 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:16:42.029245 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 20:16:42.029559 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":155,"threads_started":1}
I20260926 20:16:42.031677 9478 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41541
I20260926 20:16:42.031967 9478 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:42.032513 9478 heartbeater.cc:511] Master 127.31.119.254:41541 requested a full tablet report, sending...
I20260926 20:16:42.033749 9028 ts_manager.cc:194] Registered new tserver with Master: 6e4f50199eae464e8eb9bb34b5e8e035 (127.31.119.195:39349)
I20260926 20:16:42.034561 9028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.195:50765
I20260926 20:16:42.035336 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:16:42.038630 9067 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:42.037663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.31.119.254:41541)
0926 20:16:42.037848 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:42.037851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:42.038034 (+ 183us) server_negotiation.cc:407] Connection header received
0926 20:16:42.038106 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:42.038108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:42.038148 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:42.038222 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:42.038500 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49874: BlockingRecv error: recv got EOF from 127.0.0.1:49874 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:16:42.039103 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 9088
I20260926 20:16:42.045392 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 9219
I20260926 20:16:42.050843 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 9350
I20260926 20:16:42.056346 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 8997
2026-09-26T20:16:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (944 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T20:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:42Z Disabled control of system clock
I20260926 20:16:42.086807 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38565
--webserver_interface=127.31.119.254
--webserver_port=0
--builtin_ntp_servers=127.31.119.212:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:38565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:16:42.209508 9489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:42.209798 9489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:42.209838 9489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:42.214030 9489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:42.214105 9489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:42.214121 9489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:42.214139 9489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:42.214155 9489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:42.219205 9489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:38565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:42.220325 9489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:42.221769 9489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:42.230728 9497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:42.230710 9494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:42.230710 9495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:42.230772 9489 server_base.cc:1034] running on GCE node
I20260926 20:16:42.231628 9489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:42.232874 9489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:42.234064 9489 hybrid_clock.cc:648] HybridClock initialized: now 1790453802234026 us; error 51 us; skew 500 ppm
I20260926 20:16:42.236474 9489 webserver.cc:460] Webserver started at http://127.31.119.254:36631/ using document root <none> and password file <none>
I20260926 20:16:42.237141 9489 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:42.237250 9489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:42.237504 9489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:42.239655 9489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "10361030f7eb4579855c3dd2111ff7b6"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "d4a3674d01b5b0a81dc884dabc1f62fa"
server_key_iv: "b7f281c28005b9601f077459da87bc0d"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.240257 9489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "10361030f7eb4579855c3dd2111ff7b6"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "d4a3674d01b5b0a81dc884dabc1f62fa"
server_key_iv: "b7f281c28005b9601f077459da87bc0d"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.244637 9489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:16:42.247452 9503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.248749 9489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:16:42.248911 9489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "10361030f7eb4579855c3dd2111ff7b6"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "d4a3674d01b5b0a81dc884dabc1f62fa"
server_key_iv: "b7f281c28005b9601f077459da87bc0d"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.249049 9489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:42.254141 9489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:42.254812 9489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:42.255017 9489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:42.262117 9489 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:38565
I20260926 20:16:42.262118 9555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:38565 every 8 connection(s)
I20260926 20:16:42.263171 9489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:42.266896 9556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:42.272822 9556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6: Bootstrap starting.
I20260926 20:16:42.273554 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 9489
I20260926 20:16:42.273713 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:42.274094 32223 external_mini_cluster.cc:1428] Setting key fe894d672b9f9a8237e2aef0963548d0
I20260926 20:16:42.275872 9556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:42.276683 9556 log.cc:824] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:42.279105 9556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6: No bootstrap required, opened a new log
I20260926 20:16:42.280365 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:42.275303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.31.119.254:38565)
0926 20:16:42.276081 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:42.276100 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:42.276141 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:16:42.277116 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:42.277138 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:42.277556 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:42.277957 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:42.278821 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:42.278825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:42.278839 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:42.278849 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:42.278907 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:42.278925 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:42.279063 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:42.279268 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 20:16:42.279552 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":132,"threads_started":1}
I20260926 20:16:42.283357 9556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10361030f7eb4579855c3dd2111ff7b6" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38565 } }
I20260926 20:16:42.283639 9556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:42.283735 9556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10361030f7eb4579855c3dd2111ff7b6, State: Initialized, Role: FOLLOWER
I20260926 20:16:42.284188 9556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10361030f7eb4579855c3dd2111ff7b6" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38565 } }
I20260926 20:16:42.284360 9556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:42.284462 9556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:42.284586 9556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:42.285781 9556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10361030f7eb4579855c3dd2111ff7b6" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38565 } }
I20260926 20:16:42.286173 9556 leader_election.cc:302] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10361030f7eb4579855c3dd2111ff7b6; no voters:
I20260926 20:16:42.286542 9556 leader_election.cc:288] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:42.286684 9561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:42.286934 9561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [term 1 LEADER]: Becoming Leader. State: Replica: 10361030f7eb4579855c3dd2111ff7b6, State: Running, Role: LEADER
I20260926 20:16:42.287441 9561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10361030f7eb4579855c3dd2111ff7b6" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38565 } }
I20260926 20:16:42.287884 9556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:42.289212 9562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10361030f7eb4579855c3dd2111ff7b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10361030f7eb4579855c3dd2111ff7b6" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38565 } } }
I20260926 20:16:42.289420 9562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:42.289451 9563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10361030f7eb4579855c3dd2111ff7b6. Latest consensus state: current_term: 1 leader_uuid: "10361030f7eb4579855c3dd2111ff7b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10361030f7eb4579855c3dd2111ff7b6" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38565 } } }
I20260926 20:16:42.289579 9563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:42.290043 9571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:42.294353 9571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:42.300686 9571 catalog_manager.cc:1347] Generated new cluster ID: a80568e892f64e15b432d236e39caff9
I20260926 20:16:42.300809 9571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:42.323519 9571 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:42.323887 9571 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:42.339738 9571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10361030f7eb4579855c3dd2111ff7b6: Generated new TSK 0
I20260926 20:16:42.340555 9571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:16:42.351044 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38565
--builtin_ntp_servers=127.31.119.212:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:16:42.466784 9580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:42.467072 9580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:42.467114 9580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:42.471259 9580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:42.471323 9580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:42.471422 9580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:42.476327 9580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:42.477550 9580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:42.478962 9580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:42.488739 9586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:42.488746 9588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:42.489068 9580 server_base.cc:1034] running on GCE node
W20260926 20:16:42.488747 9585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:42.489645 9580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:42.490242 9580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:42.491443 9580 hybrid_clock.cc:648] HybridClock initialized: now 1790453802491425 us; error 31 us; skew 500 ppm
I20260926 20:16:42.493264 9580 webserver.cc:460] Webserver started at http://127.31.119.193:41729/ using document root <none> and password file <none>
I20260926 20:16:42.493888 9580 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:42.493983 9580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:42.494192 9580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:42.496362 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "6fcf4b03396047b8b126325cb4ebf7b6"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "f4461dae9d4c8acbfeb28bf6cb22bdc3"
server_key_iv: "0370f72063df6d2cfa12090572126582"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.496927 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "6fcf4b03396047b8b126325cb4ebf7b6"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "f4461dae9d4c8acbfeb28bf6cb22bdc3"
server_key_iv: "0370f72063df6d2cfa12090572126582"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.500849 9580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 20:16:42.503501 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.504706 9580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:16:42.504854 9580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "6fcf4b03396047b8b126325cb4ebf7b6"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "f4461dae9d4c8acbfeb28bf6cb22bdc3"
server_key_iv: "0370f72063df6d2cfa12090572126582"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.504976 9580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:42.509683 9580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:42.510327 9580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:42.510509 9580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:42.511034 9580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:42.511922 9580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:42.511989 9580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.512058 9580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:42.512109 9580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.521559 9580 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:42623
I20260926 20:16:42.521596 9707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:42623 every 8 connection(s)
I20260926 20:16:42.522573 9580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:42.527805 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 9580
I20260926 20:16:42.527947 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:16:42.528258 32223 external_mini_cluster.cc:1428] Setting key de6c3784b766a0e1d498a1dce10897e9
I20260926 20:16:42.529862 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:42.524731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:38661 (local address 127.31.119.254:38565)
0926 20:16:42.524930 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:42.524935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:42.526221 (+ 1286us) server_negotiation.cc:407] Connection header received
0926 20:16:42.527649 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:42.527652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:42.527695 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:42.527778 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:42.529226 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:42.529230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:42.529235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:42.529252 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:42.529273 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:42.529406 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:42.529522 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:42.529677 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 20:16:42.529726 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:16:42.530435 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38565
--builtin_ntp_servers=127.31.119.212:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 20:16:42.530922 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:42.525219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38565 (local address 127.31.119.193:38661)
0926 20:16:42.526021 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:42.526065 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:42.527364 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:42.528024 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:42.528036 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:42.528179 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:42.528928 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:42.528930 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:42.528975 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:42.529056 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:42.529508 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:42.529518 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:42.529524 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:16:42.529731 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 20:16:42.530041 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":130,"threads_started":1}
I20260926 20:16:42.532330 9708 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38565
I20260926 20:16:42.532629 9708 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:42.533164 9708 heartbeater.cc:511] Master 127.31.119.254:38565 requested a full tablet report, sending...
I20260926 20:16:42.534967 9520 ts_manager.cc:194] Registered new tserver with Master: 6fcf4b03396047b8b126325cb4ebf7b6 (127.31.119.193:42623)
W20260926 20:16:42.650080 9711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:42.650421 9711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:42.650492 9711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:42.654780 9711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:42.654872 9711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:42.654964 9711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:42.659970 9711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:42.661187 9711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:42.662716 9711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:42.672962 9719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:42.672986 9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:42.672966 9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:42.673421 9711 server_base.cc:1034] running on GCE node
I20260926 20:16:42.673925 9711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:42.674611 9711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:42.675832 9711 hybrid_clock.cc:648] HybridClock initialized: now 1790453802675816 us; error 39 us; skew 500 ppm
I20260926 20:16:42.678197 9711 webserver.cc:460] Webserver started at http://127.31.119.194:37029/ using document root <none> and password file <none>
I20260926 20:16:42.678825 9711 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:42.678925 9711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:42.679177 9711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:42.681445 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "f19e6f4e96e340ddb53d8aa355284b0b"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "8dd864b939dd611d3d802b12c23f209e"
server_key_iv: "d59950c0ab22cf7bfad418357a5412f1"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.682062 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "f19e6f4e96e340ddb53d8aa355284b0b"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "8dd864b939dd611d3d802b12c23f209e"
server_key_iv: "d59950c0ab22cf7bfad418357a5412f1"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.686197 9711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:42.689247 9725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.690855 9711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:42.690999 9711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "f19e6f4e96e340ddb53d8aa355284b0b"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "8dd864b939dd611d3d802b12c23f209e"
server_key_iv: "d59950c0ab22cf7bfad418357a5412f1"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.691140 9711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:42.696147 9711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:42.696820 9711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:42.697019 9711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:42.697674 9711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:42.698632 9711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:42.698704 9711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.698768 9711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:42.698817 9711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.708170 9711 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:41171
I20260926 20:16:42.708199 9838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:41171 every 8 connection(s)
I20260926 20:16:42.709177 9711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:42.716279 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:42.711331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:50437 (local address 127.31.119.254:38565)
0926 20:16:42.711513 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:42.711518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:42.712653 (+ 1135us) server_negotiation.cc:407] Connection header received
0926 20:16:42.714240 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:42.714244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:42.714295 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:42.714400 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:42.715724 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:42.715728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:42.715731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:42.715747 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:42.715771 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:42.715788 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:42.715898 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:42.716066 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 20:16:42.716120 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:16:42.717377 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:42.711711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38565 (local address 127.31.119.194:50437)
0926 20:16:42.712483 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:42.712529 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:42.713987 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:42.714603 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:42.714611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:42.714717 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:42.715417 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:42.715419 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:42.715472 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:42.715551 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:42.715887 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:42.715896 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:42.715904 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:16:42.716109 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 20:16:42.716391 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":132,"threads_started":1}
I20260926 20:16:42.717908 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 9711
I20260926 20:16:42.718041 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:42.718381 32223 external_mini_cluster.cc:1428] Setting key a7f24e9313f74b3717aa0138e8150ab4
I20260926 20:16:42.718845 9839 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38565
I20260926 20:16:42.719112 9839 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:42.719607 9839 heartbeater.cc:511] Master 127.31.119.254:38565 requested a full tablet report, sending...
I20260926 20:16:42.720445 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38565
--builtin_ntp_servers=127.31.119.212:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 20:16:42.720863 9520 ts_manager.cc:194] Registered new tserver with Master: f19e6f4e96e340ddb53d8aa355284b0b (127.31.119.194:41171)
W20260926 20:16:42.836520 9842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:42.836799 9842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:42.836879 9842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:42.840996 9842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:42.841061 9842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:42.841156 9842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:42.846145 9842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:42.847215 9842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:42.848565 9842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:42.858354 9847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:42.858353 9848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:42.858546 9842 server_base.cc:1034] running on GCE node
W20260926 20:16:42.858429 9850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:42.859107 9842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:42.859766 9842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:42.860931 9842 hybrid_clock.cc:648] HybridClock initialized: now 1790453802860913 us; error 30 us; skew 500 ppm
I20260926 20:16:42.862973 9842 webserver.cc:460] Webserver started at http://127.31.119.195:41649/ using document root <none> and password file <none>
I20260926 20:16:42.863649 9842 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:42.863713 9842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:42.863987 9842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:42.866200 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "f0c80d174d8e49afb6c2122fbb5a2ee4"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "22fa5e1eca4e96dd5e1d3c3480b91582"
server_key_iv: "ab8131c0e5290ad69f616cd22964595e"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.866777 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "f0c80d174d8e49afb6c2122fbb5a2ee4"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "22fa5e1eca4e96dd5e1d3c3480b91582"
server_key_iv: "ab8131c0e5290ad69f616cd22964595e"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.870687 9842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:16:42.873433 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.874778 9842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:16:42.874944 9842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "f0c80d174d8e49afb6c2122fbb5a2ee4"
format_stamp: "Formatted at 2026-09-26 20:16:42 on dist-test-slave-vnbh"
server_key: "22fa5e1eca4e96dd5e1d3c3480b91582"
server_key_iv: "ab8131c0e5290ad69f616cd22964595e"
server_key_version: "encryptionkey@0"
I20260926 20:16:42.875082 9842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:42.879765 9842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:42.880414 9842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:42.880602 9842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:42.881140 9842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:42.882117 9842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:42.882190 9842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.882262 9842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:42.882308 9842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:42.891738 9842 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:45627
I20260926 20:16:42.891758 9969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:45627 every 8 connection(s)
I20260926 20:16:42.892700 9842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:42.897061 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 9842
I20260926 20:16:42.897193 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:42.897559 32223 external_mini_cluster.cc:1428] Setting key 08d07434e064bcf77437161eaa933fa8
I20260926 20:16:42.899770 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:42.894763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:56443 (local address 127.31.119.254:38565)
0926 20:16:42.894893 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:42.894897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:42.896040 (+ 1143us) server_negotiation.cc:407] Connection header received
0926 20:16:42.897796 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:42.897799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:42.897849 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:42.897967 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:42.899236 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:42.899240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:42.899243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:16:42.899255 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:42.899280 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:16:42.899293 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:42.899449 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:42.899595 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 20:16:42.899642 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:16:42.900672 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:42.895177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38565 (local address 127.31.119.195:56443)
0926 20:16:42.895871 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:42.895912 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:42.897493 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:42.898106 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:42.898114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:42.898234 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:42.898945 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:16:42.898949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:42.899014 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:16:42.899087 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:42.899436 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:42.899444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:42.899451 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:16:42.899643 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 20:16:42.899908 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":125,"threads_started":1}
I20260926 20:16:42.901877 9970 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38565
I20260926 20:16:42.902136 9970 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:42.902642 9970 heartbeater.cc:511] Master 127.31.119.254:38565 requested a full tablet report, sending...
I20260926 20:16:42.903678 9520 ts_manager.cc:194] Registered new tserver with Master: f0c80d174d8e49afb6c2122fbb5a2ee4 (127.31.119.195:45627)
I20260926 20:16:42.911723 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:16:42.914830 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:42.913798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.31.119.254:38565)
0926 20:16:42.913960 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:42.913963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:42.914160 (+ 197us) server_negotiation.cc:407] Connection header received
0926 20:16:42.914298 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:42.914302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:42.914339 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:42.914407 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:42.914605 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53438: BlockingRecv error: recv got EOF from 127.0.0.1:53438 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:16:42.915014 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 9580
I20260926 20:16:42.920612 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 9711
I20260926 20:16:42.926359 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 9842
I20260926 20:16:42.931479 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 9489
2026-09-26T20:16:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (874 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:16:42 dist-test-slave-vnbh krb5kdc[9980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:16:42 dist-test-slave-vnbh krb5kdc[9980](info): set up 2 sockets
Sep 26 20:16:42 dist-test-slave-vnbh krb5kdc[9980](info): commencing operation
krb5kdc: starting...
W20260926 20:16:43.975164 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:44 dist-test-slave-vnbh krb5kdc[9980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:16:44.100950 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:34133
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:34133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:44.214869 9993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:44.215210 9993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:44.215293 9993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:44.219499 9993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:16:44.219619 9993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:44.219657 9993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:44.219698 9993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:16:44.219729 9993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:16:44.225420 9993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:34133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:34133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:44.226701 9993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:44.227793 9993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:44.237112 9999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:44.237128 9993 server_base.cc:1034] running on GCE node
W20260926 20:16:44.237149 10001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:44.237151 9998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:44.237982 9993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:44.239199 9993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:44.240386 9993 hybrid_clock.cc:648] HybridClock initialized: now 1790453804240350 us; error 51 us; skew 500 ppm
Sep 26 20:16:44 dist-test-slave-vnbh krb5kdc[9980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453804, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:44.243675 9993 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:16:44.244602 9993 webserver.cc:460] Webserver started at http://127.31.119.254:44857/ using document root <none> and password file <none>
I20260926 20:16:44.245162 9993 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:44.245234 9993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:44.245489 9993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:44.247617 9993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "fafcbe7518dd47eda6dda5af6f9f3771"
format_stamp: "Formatted at 2026-09-26 20:16:44 on dist-test-slave-vnbh"
server_key: "10ad5944f7ba2fb2c527e77ed26a2eba"
server_key_iv: "f35bbf07c8c799e66af799627b31198b"
server_key_version: "encryptionkey@0"
I20260926 20:16:44.248175 9993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "fafcbe7518dd47eda6dda5af6f9f3771"
format_stamp: "Formatted at 2026-09-26 20:16:44 on dist-test-slave-vnbh"
server_key: "10ad5944f7ba2fb2c527e77ed26a2eba"
server_key_iv: "f35bbf07c8c799e66af799627b31198b"
server_key_version: "encryptionkey@0"
I20260926 20:16:44.252193 9993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:16:44.255067 10008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:44.256610 9993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:16:44.256753 9993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "fafcbe7518dd47eda6dda5af6f9f3771"
format_stamp: "Formatted at 2026-09-26 20:16:44 on dist-test-slave-vnbh"
server_key: "10ad5944f7ba2fb2c527e77ed26a2eba"
server_key_iv: "f35bbf07c8c799e66af799627b31198b"
server_key_version: "encryptionkey@0"
I20260926 20:16:44.256865 9993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:44.290544 9993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:44.294267 9993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:44.294433 9993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:44.302062 9993 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:34133
I20260926 20:16:44.302393 10060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:34133 every 8 connection(s)
I20260926 20:16:44.303128 9993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:16:44.306429 10061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:44.308274 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 9993
I20260926 20:16:44.308380 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:16:44.308779 32223 external_mini_cluster.cc:1428] Setting key 3a87736edd900598ef0dcd54f8400490
I20260926 20:16:44.313553 10061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771: Bootstrap starting.
I20260926 20:16:44.315990 10061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:44.316784 10061 log.cc:824] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:16:44 dist-test-slave-vnbh krb5kdc[9980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:44.319005 10061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771: No bootstrap required, opened a new log
I20260926 20:16:44.322662 10061 raft_consensus.cc:348] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcbe7518dd47eda6dda5af6f9f3771" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34133 } }
I20260926 20:16:44.322983 10061 raft_consensus.cc:374] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:44.323078 10061 raft_consensus.cc:729] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fafcbe7518dd47eda6dda5af6f9f3771, State: Initialized, Role: FOLLOWER
I20260926 20:16:44.323539 10061 consensus_queue.cc:260] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcbe7518dd47eda6dda5af6f9f3771" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34133 } }
I20260926 20:16:44.323671 10061 raft_consensus.cc:388] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:16:44.323761 10061 raft_consensus.cc:482] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:16:44.323860 10061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:44.324090 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:44.310380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.31.119.254:34133)
0926 20:16:44.310848 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:44.310864 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:44.310928 (+ 64us) server_negotiation.cc:407] Connection header received
0926 20:16:44.311674 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:44.311696 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:44.312078 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:44.312448 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:44.313337 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:44.314395 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:44.315290 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:44.315868 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:44.316061 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:44.318764 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:44.318769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:44.318783 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:44.318814 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:44.320691 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:44.321193 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:44.321198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:44.321203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:44.321390 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:44.321692 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:44.321697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:44.321699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:44.322136 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:44.322375 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:44.322686 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 20:16:44.323056 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":134,"threads_started":1}
I20260926 20:16:44.324983 10061 raft_consensus.cc:504] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcbe7518dd47eda6dda5af6f9f3771" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34133 } }
I20260926 20:16:44.325435 10061 leader_election.cc:302] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fafcbe7518dd47eda6dda5af6f9f3771; no voters:
I20260926 20:16:44.325809 10061 leader_election.cc:288] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:16:44.325896 10066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:44.326102 10066 raft_consensus.cc:686] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [term 1 LEADER]: Becoming Leader. State: Replica: fafcbe7518dd47eda6dda5af6f9f3771, State: Running, Role: LEADER
I20260926 20:16:44.326481 10066 consensus_queue.cc:237] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcbe7518dd47eda6dda5af6f9f3771" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34133 } }
I20260926 20:16:44.327188 10061 sys_catalog.cc:564] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:16:44.327795 10066 sys_catalog.cc:455] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fafcbe7518dd47eda6dda5af6f9f3771. Latest consensus state: current_term: 1 leader_uuid: "fafcbe7518dd47eda6dda5af6f9f3771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcbe7518dd47eda6dda5af6f9f3771" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34133 } } }
I20260926 20:16:44.327924 10066 sys_catalog.cc:458] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:44.328222 10067 sys_catalog.cc:455] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fafcbe7518dd47eda6dda5af6f9f3771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcbe7518dd47eda6dda5af6f9f3771" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34133 } } }
I20260926 20:16:44.328642 10067 sys_catalog.cc:458] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771 [sys.catalog]: This master's current role is: LEADER
I20260926 20:16:44.328807 10072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:16:44.332063 10072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:16:44.337983 10072 catalog_manager.cc:1347] Generated new cluster ID: c19392ea8ea2486da31a19ed5641b67a
I20260926 20:16:44.338088 10072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:16:44.380427 10072 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:16:44.381446 10072 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:16:44.385147 10072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771: Generated new TSK 0
I20260926 20:16:44.385856 10072 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:16:44.421398 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34133
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:44.555146 10087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:44.555444 10087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:44.555490 10087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:44.559518 10087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:44.559580 10087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:44.559639 10087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:16:44.567693 10087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:44.569451 10087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:44.570988 10087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:44.582573 10093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:44.582505 10092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:44.582470 10095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:44.582835 10087 server_base.cc:1034] running on GCE node
I20260926 20:16:44.583343 10087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:44.584040 10087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:44.585258 10087 hybrid_clock.cc:648] HybridClock initialized: now 1790453804585216 us; error 55 us; skew 500 ppm
Sep 26 20:16:44 dist-test-slave-vnbh krb5kdc[9980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453804, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:44.588425 10087 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:16:44.589521 10087 webserver.cc:460] Webserver started at http://127.31.119.193:44347/ using document root <none> and password file <none>
I20260926 20:16:44.590191 10087 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:44.590293 10087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:44.590539 10087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:44.592800 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "b5e31e64795d48fd98784565b108a43a"
format_stamp: "Formatted at 2026-09-26 20:16:44 on dist-test-slave-vnbh"
server_key: "6938f3c8db1afbe97aa65d8e8c694ed1"
server_key_iv: "62079dc8617e722fe1577c262efccb8b"
server_key_version: "encryptionkey@0"
I20260926 20:16:44.593475 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "b5e31e64795d48fd98784565b108a43a"
format_stamp: "Formatted at 2026-09-26 20:16:44 on dist-test-slave-vnbh"
server_key: "6938f3c8db1afbe97aa65d8e8c694ed1"
server_key_iv: "62079dc8617e722fe1577c262efccb8b"
server_key_version: "encryptionkey@0"
I20260926 20:16:44.597637 10087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:16:44.600425 10102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:44.602119 10087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:44.602283 10087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "b5e31e64795d48fd98784565b108a43a"
format_stamp: "Formatted at 2026-09-26 20:16:44 on dist-test-slave-vnbh"
server_key: "6938f3c8db1afbe97aa65d8e8c694ed1"
server_key_iv: "62079dc8617e722fe1577c262efccb8b"
server_key_version: "encryptionkey@0"
I20260926 20:16:44.602417 10087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:44.613482 10087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:44.617029 10087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:44.617223 10087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:44.617887 10087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:44.618848 10087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:44.618902 10087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:44.618948 10087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:44.618970 10087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:44.628958 10087 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:43583
I20260926 20:16:44.629171 10215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:43583 every 8 connection(s)
I20260926 20:16:44.630055 10087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:16:44.630748 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 10087
I20260926 20:16:44.630862 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:16:44.631155 32223 external_mini_cluster.cc:1428] Setting key 4312d9e2f130d1c3508c77a4a64364fb
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
Sep 26 20:16:44 dist-test-slave-vnbh krb5kdc[9980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453804, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:44.650707 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:44.632157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:36741 (local address 127.31.119.254:34133)
0926 20:16:44.632301 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:44.632306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:44.633630 (+ 1324us) server_negotiation.cc:407] Connection header received
0926 20:16:44.635445 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:44.635450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:44.635515 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:44.635617 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:44.637465 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:44.638056 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:44.640078 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:44.640328 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:44.640428 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:44.646100 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:44.646104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:44.646106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:44.646136 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:44.648039 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:44.648802 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:44.648806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:44.648807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:44.648897 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:44.649368 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:44.649370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:44.649371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:44.649569 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:44.649647 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:44.650445 (+ 798us) server_negotiation.cc:299] Negotiation successful
0926 20:16:44.650577 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:16:44.652120 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:44.632777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:34133 (local address 127.31.119.193:36741)
0926 20:16:44.633379 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:44.633508 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:44.635194 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:44.635794 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:44.635802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:44.636361 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:44.637183 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:44.637209 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:44.638205 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:44.638211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:44.639909 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:44.639926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:44.640516 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:44.640523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:44.640759 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:44.641604 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:44.641633 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:44.645863 (+ 4230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:44.648196 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:44.648206 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:44.648229 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:44.648671 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:44.648993 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:44.648999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:44.649004 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:44.649215 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:44.649684 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:44.649694 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:44.650116 (+ 422us) client_negotiation.cc:769] Sending connection context
0926 20:16:44.650367 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 20:16:44.650799 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":109,"threads_started":1}
I20260926 20:16:44.653581 10216 heartbeater.cc:347] Connected to a master server at 127.31.119.254:34133
I20260926 20:16:44.653853 10216 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:44.654390 10216 heartbeater.cc:511] Master 127.31.119.254:34133 requested a full tablet report, sending...
I20260926 20:16:44.656070 10025 ts_manager.cc:194] Registered new tserver with Master: b5e31e64795d48fd98784565b108a43a (127.31.119.193:43583)
I20260926 20:16:44.657224 10025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:36741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:16:44.664994 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34133
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:44.780315 10221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:44.780608 10221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:44.780689 10221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:44.784964 10221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:44.785032 10221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:44.785137 10221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:16:44.790606 10221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:44.791694 10221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:44.792758 10221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:44.802639 10227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:44.802734 10226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:44.802668 10229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:44.802783 10221 server_base.cc:1034] running on GCE node
I20260926 20:16:44.803396 10221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:44.804121 10221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:44.805356 10221 hybrid_clock.cc:648] HybridClock initialized: now 1790453804805328 us; error 33 us; skew 500 ppm
Sep 26 20:16:44 dist-test-slave-vnbh krb5kdc[9980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453804, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:44.808342 10221 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:16:44.809199 10221 webserver.cc:460] Webserver started at http://127.31.119.194:41725/ using document root <none> and password file <none>
I20260926 20:16:44.809847 10221 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:44.809909 10221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:44.810083 10221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:44.812120 10221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "3a6608fef39148209bdf7f7f8caad452"
format_stamp: "Formatted at 2026-09-26 20:16:44 on dist-test-slave-vnbh"
server_key: "a2a0c87c9e3c5f0c7f150d3719884bff"
server_key_iv: "99b73b6a52862451aab6c9131beacf00"
server_key_version: "encryptionkey@0"
I20260926 20:16:44.812631 10221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "3a6608fef39148209bdf7f7f8caad452"
format_stamp: "Formatted at 2026-09-26 20:16:44 on dist-test-slave-vnbh"
server_key: "a2a0c87c9e3c5f0c7f150d3719884bff"
server_key_iv: "99b73b6a52862451aab6c9131beacf00"
server_key_version: "encryptionkey@0"
I20260926 20:16:44.816573 10221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:16:44.819226 10236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:44.820580 10221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:16:44.820716 10221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "3a6608fef39148209bdf7f7f8caad452"
format_stamp: "Formatted at 2026-09-26 20:16:44 on dist-test-slave-vnbh"
server_key: "a2a0c87c9e3c5f0c7f150d3719884bff"
server_key_iv: "99b73b6a52862451aab6c9131beacf00"
server_key_version: "encryptionkey@0"
I20260926 20:16:44.820842 10221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:44.835959 10221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:44.839095 10221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:44.839299 10221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:44.839910 10221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:44.840739 10221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:44.840782 10221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:44.840845 10221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:44.840878 10221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:44.849999 10221 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:45103
I20260926 20:16:44.850016 10349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:45103 every 8 connection(s)
I20260926 20:16:44.850935 10221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:16:44.851717 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 10221
I20260926 20:16:44.851840 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:16:44.852146 32223 external_mini_cluster.cc:1428] Setting key 888ae256b4167526553f271d33a261d5
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
Sep 26 20:16:44 dist-test-slave-vnbh krb5kdc[9980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453804, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:44.871140 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:44.853042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:44859 (local address 127.31.119.254:34133)
0926 20:16:44.853187 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:44.853192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:44.854252 (+ 1060us) server_negotiation.cc:407] Connection header received
0926 20:16:44.856358 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:44.856363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:44.856440 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:44.856552 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:44.857964 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:44.858890 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:44.860169 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:44.860539 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:44.860660 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:44.866648 (+ 5988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:44.866652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:44.866654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:44.866682 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:44.868615 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:44.869180 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:44.869186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:44.869191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:44.869311 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:44.869737 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:44.869744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:44.869748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:44.870012 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:44.870118 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:44.870738 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 20:16:44.870897 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:16:44.871861 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:44.853517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:34133 (local address 127.31.119.194:44859)
0926 20:16:44.854072 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:44.854130 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:44.856103 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:44.856734 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:44.856743 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:44.857135 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:44.857799 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:44.857814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:44.859036 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:44.859039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:44.860037 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:44.860047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:44.860738 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:44.860744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:44.860928 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:44.861695 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:44.861713 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:44.866252 (+ 4539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:44.868747 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:44.868752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:44.868768 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:44.869050 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:44.869405 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:44.869407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:44.869408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:44.869620 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:44.870122 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:44.870133 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:44.870457 (+ 324us) client_negotiation.cc:769] Sending connection context
0926 20:16:44.870619 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 20:16:44.870895 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":136,"threads_started":1}
I20260926 20:16:44.873152 10350 heartbeater.cc:347] Connected to a master server at 127.31.119.254:34133
I20260926 20:16:44.873422 10350 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:44.873925 10350 heartbeater.cc:511] Master 127.31.119.254:34133 requested a full tablet report, sending...
I20260926 20:16:44.875012 10025 ts_manager.cc:194] Registered new tserver with Master: 3a6608fef39148209bdf7f7f8caad452 (127.31.119.194:45103)
I20260926 20:16:44.875699 10025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:44859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:16:44.884564 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34133
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:16:44.999935 10355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:16:45.000258 10355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:16:45.000337 10355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:16:45.004556 10355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:16:45.004674 10355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:16:45.004782 10355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:16:45.010859 10355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:16:45.012090 10355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:16:45.013211 10355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:16:45.023577 10363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:16:45.023597 10360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:45.023620 10355 server_base.cc:1034] running on GCE node
W20260926 20:16:45.023577 10361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:16:45.024435 10355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:16:45.025252 10355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:16:45.026496 10355 hybrid_clock.cc:648] HybridClock initialized: now 1790453805026458 us; error 52 us; skew 500 ppm
Sep 26 20:16:45 dist-test-slave-vnbh krb5kdc[9980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:16:45.029844 10355 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:16:45.030826 10355 webserver.cc:460] Webserver started at http://127.31.119.195:35127/ using document root <none> and password file <none>
I20260926 20:16:45.031384 10355 fs_manager.cc:359] Metadata directory not provided
I20260926 20:16:45.031440 10355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:16:45.031605 10355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:16:45.033738 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "3f68ccd937954663b0e9637e1aa57615"
format_stamp: "Formatted at 2026-09-26 20:16:45 on dist-test-slave-vnbh"
server_key: "32ce9bb68899b2fdeb85f7cd86183aa2"
server_key_iv: "f26356cfdc3cea09b3fe7abb32d9cfe4"
server_key_version: "encryptionkey@0"
I20260926 20:16:45.034283 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "3f68ccd937954663b0e9637e1aa57615"
format_stamp: "Formatted at 2026-09-26 20:16:45 on dist-test-slave-vnbh"
server_key: "32ce9bb68899b2fdeb85f7cd86183aa2"
server_key_iv: "f26356cfdc3cea09b3fe7abb32d9cfe4"
server_key_version: "encryptionkey@0"
I20260926 20:16:45.038264 10355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:16:45.040964 10370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:45.042606 10355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:16:45.042773 10355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "3f68ccd937954663b0e9637e1aa57615"
format_stamp: "Formatted at 2026-09-26 20:16:45 on dist-test-slave-vnbh"
server_key: "32ce9bb68899b2fdeb85f7cd86183aa2"
server_key_iv: "f26356cfdc3cea09b3fe7abb32d9cfe4"
server_key_version: "encryptionkey@0"
I20260926 20:16:45.042915 10355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:16:45.054951 10355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:16:45.058569 10355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:16:45.058790 10355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:16:45.059419 10355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:16:45.060361 10355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:16:45.060441 10355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:45.060524 10355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:16:45.060559 10355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:16:45.070076 10355 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:41503
I20260926 20:16:45.070096 10483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:41503 every 8 connection(s)
I20260926 20:16:45.071002 10355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:16:45.071406 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 10355
I20260926 20:16:45.071560 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:16:45.071892 32223 external_mini_cluster.cc:1428] Setting key 18e4b19ca2b398d7c1afdde7ac321088
Sep 26 20:16:45 dist-test-slave-vnbh krb5kdc[9980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:16:45.085165 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.072996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:58389 (local address 127.31.119.254:34133)
0926 20:16:45.073159 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:45.073163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:45.074171 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 20:16:45.075597 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:45.075601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:45.075653 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:45.075730 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:45.076913 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.077506 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.078650 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.078875 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.078989 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.081585 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:45.081588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:45.081590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:45.081618 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:45.082983 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:45.083554 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:45.083563 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:45.083564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:45.083645 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:45.084030 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:45.084035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:45.084037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:45.084226 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:45.084346 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:45.084873 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 20:16:45.085000 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:16:45.086275 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.073560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:34133 (local address 127.31.119.195:58389)
0926 20:16:45.074030 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:45.074081 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:45.075390 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:45.075902 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:45.075908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:45.076284 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:16:45.076766 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.076779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.077635 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.077638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.078527 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.078536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.079028 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.079032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.079205 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.079776 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:16:45.079792 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:16:45.081384 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:16:45.083110 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:45.083116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:45.083133 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:45.083435 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:45.083739 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:16:45.083743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:16:45.083745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:16:45.083902 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:16:45.084342 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:16:45.084349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:16:45.084610 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 20:16:45.084826 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 20:16:45.085166 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
I20260926 20:16:45.087441 10484 heartbeater.cc:347] Connected to a master server at 127.31.119.254:34133
I20260926 20:16:45.087714 10484 heartbeater.cc:464] Registering TS with master...
I20260926 20:16:45.088243 10484 heartbeater.cc:511] Master 127.31.119.254:34133 requested a full tablet report, sending...
I20260926 20:16:45.089540 10025 ts_manager.cc:194] Registered new tserver with Master: 3f68ccd937954663b0e9637e1aa57615 (127.31.119.195:41503)
I20260926 20:16:45.090094 10025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:58389
I20260926 20:16:45.098292 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:16:45.108336 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.100233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.31.119.254:34133)
0926 20:16:45.100420 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:45.100425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:45.100603 (+ 178us) server_negotiation.cc:407] Connection header received
0926 20:16:45.100760 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:45.100764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:45.100813 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:45.100931 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:16:45.101731 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.102262 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.103123 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.103366 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.103476 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.104949 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:16:45.104952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:16:45.104955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:16:45.104982 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:16:45.106500 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:45.106977 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:45.106980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:45.106982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:45.107072 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:16:45.107367 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:16:45.107370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:16:45.107372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:16:45.107661 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:16:45.107797 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:45.108047 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:16:45.108167 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 20:16:45.111807 10025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:16:45.115010 10025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:16:45.130995 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.124800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.31.119.193:43583)
0926 20:16:45.125177 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:45.125193 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:45.125236 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:16:45.125477 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:45.125484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:45.125704 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:45.126275 (+ 571us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:45.126441 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.127636 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.129509 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.130232 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.130379 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.130517 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:45.130720 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 20:16:45.130829 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":76,"threads_started":1}
I20260926 20:16:45.130985 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.124629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:43583 (local address 127.0.0.1:53966)
0926 20:16:45.125047 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:45.125108 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:45.125260 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:45.125920 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:45.125929 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:45.125951 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:45.126113 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.126120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.127777 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.127780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.129364 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.129404 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.130385 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.130389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.130501 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.130544 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:16:45.130665 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 20:16:45.130790 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1,"thread_start_us":141,"threads_started":1}
I20260926 20:16:45.131383 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.125478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:41503 (local address 127.0.0.1:40140)
0926 20:16:45.125802 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:45.125818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:45.125895 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:45.126684 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:45.126687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:45.126703 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:45.126818 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.126828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.128395 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.128399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.129922 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.129932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.130971 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.130974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.131081 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.131105 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:16:45.131159 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:16:45.131239 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":150,"thread_start_us":96,"threads_started":1}
I20260926 20:16:45.131538 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.125801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:45103 (local address 127.0.0.1:45092)
0926 20:16:45.126398 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:45.126411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:45.126506 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:45.127002 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:45.127005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:45.127023 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:45.127118 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.127125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.128433 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.128437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.129858 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.129870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.131145 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.131151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.131241 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.131262 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:16:45.131327 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:16:45.131395 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":43,"thread_start_us":97,"threads_started":1}
I20260926 20:16:45.131974 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.126177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.31.119.194:45103)
0926 20:16:45.126588 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:45.126599 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:45.126622 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:16:45.126691 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:45.126696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:45.126872 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:45.127020 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:45.127214 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.128245 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.130000 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.130823 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.130943 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.131071 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:45.131709 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 20:16:45.131820 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":112,"threads_started":1}
I20260926 20:16:45.132237 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.125755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.31.119.195:41503)
0926 20:16:45.126205 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:45.126218 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:45.126240 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:16:45.126327 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:45.126334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:45.126531 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:45.126713 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:45.126942 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.128221 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.130054 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.130838 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.131801 (+ 963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.131941 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:45.131989 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 20:16:45.132076 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":150,"threads_started":1}
I20260926 20:16:45.134332 10150 tablet_service.cc:1467] Processing CreateTablet for tablet 12cbd0fc8d9b43128ce1565b81364c10 (DEFAULT_TABLE table=test-table [id=e94e76afdbf441c794dd088a5bac12eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:45.134784 10284 tablet_service.cc:1467] Processing CreateTablet for tablet 12cbd0fc8d9b43128ce1565b81364c10 (DEFAULT_TABLE table=test-table [id=e94e76afdbf441c794dd088a5bac12eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:45.135250 10150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12cbd0fc8d9b43128ce1565b81364c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:45.135823 10284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12cbd0fc8d9b43128ce1565b81364c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:45.138561 10418 tablet_service.cc:1467] Processing CreateTablet for tablet 12cbd0fc8d9b43128ce1565b81364c10 (DEFAULT_TABLE table=test-table [id=e94e76afdbf441c794dd088a5bac12eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:16:45.139698 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12cbd0fc8d9b43128ce1565b81364c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:16:45.141836 10501 tablet_bootstrap.cc:492] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a: Bootstrap starting.
I20260926 20:16:45.143479 10502 tablet_bootstrap.cc:492] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: Bootstrap starting.
I20260926 20:16:45.144126 10501 tablet_bootstrap.cc:654] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:45.145246 10501 log.cc:824] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:45.146257 10502 tablet_bootstrap.cc:654] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:45.147185 10502 log.cc:824] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:45.147512 10501 tablet_bootstrap.cc:492] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a: No bootstrap required, opened a new log
I20260926 20:16:45.147728 10504 tablet_bootstrap.cc:492] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: Bootstrap starting.
I20260926 20:16:45.147866 10501 ts_tablet_manager.cc:1397] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:16:45.149056 10502 tablet_bootstrap.cc:492] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: No bootstrap required, opened a new log
I20260926 20:16:45.149322 10502 ts_tablet_manager.cc:1397] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:16:45.149992 10504 tablet_bootstrap.cc:654] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: Neither blocks nor log segments found. Creating new log.
I20260926 20:16:45.151247 10504 log.cc:824] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: Log is configured to *not* fsync() on all Append() calls
I20260926 20:16:45.152103 10501 raft_consensus.cc:348] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e31e64795d48fd98784565b108a43a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 43583 } } peers { permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 } } peers { permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 } }
I20260926 20:16:45.152384 10501 raft_consensus.cc:374] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:45.152431 10501 raft_consensus.cc:729] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e31e64795d48fd98784565b108a43a, State: Initialized, Role: FOLLOWER
I20260926 20:16:45.152576 10502 raft_consensus.cc:348] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e31e64795d48fd98784565b108a43a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 43583 } } peers { permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 } } peers { permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 } }
I20260926 20:16:45.152870 10502 raft_consensus.cc:374] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:45.152925 10502 raft_consensus.cc:729] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6608fef39148209bdf7f7f8caad452, State: Initialized, Role: FOLLOWER
I20260926 20:16:45.152899 10501 consensus_queue.cc:260] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e31e64795d48fd98784565b108a43a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 43583 } } peers { permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 } } peers { permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 } }
I20260926 20:16:45.153401 10502 consensus_queue.cc:260] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e31e64795d48fd98784565b108a43a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 43583 } } peers { permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 } } peers { permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 } }
I20260926 20:16:45.153808 10504 tablet_bootstrap.cc:492] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: No bootstrap required, opened a new log
I20260926 20:16:45.153959 10501 ts_tablet_manager.cc:1428] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260926 20:16:45.153986 10504 ts_tablet_manager.cc:1397] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:16:45.154337 10216 heartbeater.cc:503] Master 127.31.119.254:34133 was elected leader, sending a full tablet report...
I20260926 20:16:45.154408 10502 ts_tablet_manager.cc:1428] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:16:45.154747 10350 heartbeater.cc:503] Master 127.31.119.254:34133 was elected leader, sending a full tablet report...
I20260926 20:16:45.157481 10504 raft_consensus.cc:348] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e31e64795d48fd98784565b108a43a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 43583 } } peers { permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 } } peers { permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 } }
I20260926 20:16:45.157773 10504 raft_consensus.cc:374] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:16:45.157831 10504 raft_consensus.cc:729] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f68ccd937954663b0e9637e1aa57615, State: Initialized, Role: FOLLOWER
I20260926 20:16:45.158278 10504 consensus_queue.cc:260] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e31e64795d48fd98784565b108a43a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 43583 } } peers { permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 } } peers { permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 } }
I20260926 20:16:45.159351 10504 ts_tablet_manager.cc:1428] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:16:45.159382 10484 heartbeater.cc:503] Master 127.31.119.254:34133 was elected leader, sending a full tablet report...
I20260926 20:16:45.258639 10507 raft_consensus.cc:482] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:16:45.258890 10507 raft_consensus.cc:504] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e31e64795d48fd98784565b108a43a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 43583 } } peers { permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 } } peers { permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 } }
I20260926 20:16:45.260142 10507 leader_election.cc:288] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f68ccd937954663b0e9637e1aa57615 (127.31.119.195:41503), 3a6608fef39148209bdf7f7f8caad452 (127.31.119.194:45103)
I20260926 20:16:45.264673 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.260623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:50081 (local address 127.31.119.195:41503)
0926 20:16:45.260770 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:45.260774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:45.260998 (+ 224us) server_negotiation.cc:407] Connection header received
0926 20:16:45.261405 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:45.261412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:45.261474 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:45.261556 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:45.261838 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.262364 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.263607 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.264018 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.264149 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.264207 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:45.264452 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 20:16:45.264525 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:16:45.264683 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.261178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:55163 (local address 127.31.119.194:45103)
0926 20:16:45.261359 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:45.261365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:45.261437 (+ 72us) server_negotiation.cc:407] Connection header received
0926 20:16:45.261591 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:45.261594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:45.261649 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:45.261730 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:16:45.262006 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.262534 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.263617 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.264033 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.264166 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.264222 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:45.264473 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 20:16:45.264550 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:16:45.264715 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.261026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:45103 (local address 127.31.119.193:55163)
0926 20:16:45.261333 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:45.261348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:45.261465 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:45.261758 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:45.261761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:45.261778 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:45.261892 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.261897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.262652 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.262656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.263486 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.263496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.264159 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.264162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.264313 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.264353 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:16:45.264447 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:16:45.264524 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":48,"thread_start_us":107,"threads_started":1}
I20260926 20:16:45.264724 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.260492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:41503 (local address 127.31.119.193:50081)
0926 20:16:45.260865 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:45.260884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:16:45.261010 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:16:45.261596 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:16:45.261599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:16:45.261613 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:16:45.261738 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.261748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.262501 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.262507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.263471 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:16:45.263481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.264150 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.264153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:16:45.264254 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.264343 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 20:16:45.264436 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 20:16:45.264519 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":130,"thread_start_us":159,"threads_started":1}
I20260926 20:16:45.265801 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12cbd0fc8d9b43128ce1565b81364c10" candidate_uuid: "b5e31e64795d48fd98784565b108a43a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f68ccd937954663b0e9637e1aa57615" is_pre_election: true
I20260926 20:16:45.265837 10304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12cbd0fc8d9b43128ce1565b81364c10" candidate_uuid: "b5e31e64795d48fd98784565b108a43a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6608fef39148209bdf7f7f8caad452" is_pre_election: true
I20260926 20:16:45.266150 10304 raft_consensus.cc:2454] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5e31e64795d48fd98784565b108a43a in term 0.
I20260926 20:16:45.266157 10438 raft_consensus.cc:2454] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5e31e64795d48fd98784565b108a43a in term 0.
I20260926 20:16:45.266638 10104 leader_election.cc:302] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f68ccd937954663b0e9637e1aa57615, b5e31e64795d48fd98784565b108a43a; no voters:
I20260926 20:16:45.266937 10507 raft_consensus.cc:2781] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:16:45.267037 10507 raft_consensus.cc:482] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:16:45.267122 10507 raft_consensus.cc:3037] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:45.268217 10507 raft_consensus.cc:504] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e31e64795d48fd98784565b108a43a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 43583 } } peers { permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 } } peers { permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 } }
I20260926 20:16:45.268586 10507 leader_election.cc:288] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [CANDIDATE]: Term 1 election: Requested vote from peers 3f68ccd937954663b0e9637e1aa57615 (127.31.119.195:41503), 3a6608fef39148209bdf7f7f8caad452 (127.31.119.194:45103)
I20260926 20:16:45.268980 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12cbd0fc8d9b43128ce1565b81364c10" candidate_uuid: "b5e31e64795d48fd98784565b108a43a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f68ccd937954663b0e9637e1aa57615"
I20260926 20:16:45.269075 10304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12cbd0fc8d9b43128ce1565b81364c10" candidate_uuid: "b5e31e64795d48fd98784565b108a43a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6608fef39148209bdf7f7f8caad452"
I20260926 20:16:45.269089 10438 raft_consensus.cc:3037] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:45.269229 10304 raft_consensus.cc:3037] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:16:45.270227 10438 raft_consensus.cc:2454] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e31e64795d48fd98784565b108a43a in term 1.
I20260926 20:16:45.270257 10304 raft_consensus.cc:2454] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5e31e64795d48fd98784565b108a43a in term 1.
I20260926 20:16:45.270498 10104 leader_election.cc:302] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f68ccd937954663b0e9637e1aa57615, b5e31e64795d48fd98784565b108a43a; no voters:
I20260926 20:16:45.270668 10507 raft_consensus.cc:2781] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:16:45.270891 10507 raft_consensus.cc:686] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 1 LEADER]: Becoming Leader. State: Replica: b5e31e64795d48fd98784565b108a43a, State: Running, Role: LEADER
I20260926 20:16:45.271260 10507 consensus_queue.cc:237] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e31e64795d48fd98784565b108a43a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 43583 } } peers { permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 } } peers { permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 } }
I20260926 20:16:45.274736 10024 catalog_manager.cc:5495] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a reported cstate change: term changed from 0 to 1, leader changed from <none> to b5e31e64795d48fd98784565b108a43a (127.31.119.193). New cstate: current_term: 1 leader_uuid: "b5e31e64795d48fd98784565b108a43a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5e31e64795d48fd98784565b108a43a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 43583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 } health_report { overall_health: UNKNOWN } } }
W20260926 20:16:45.322181 10485 tablet.cc:2369] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:16:45.322682 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:16:45.318816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.31.119.193:43583)
0926 20:16:45.318973 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:16:45.318976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:16:45.318989 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:16:45.319064 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:16:45.319067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:16:45.319121 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:16:45.319211 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:16:45.319547 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.320042 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.320947 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:16:45.321156 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:16:45.321237 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:16:45.321641 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:16:45.322202 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:16:45.322413 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:16:45.322456 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 20:16:45.322510 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:16:45.329134 10438 raft_consensus.cc:1261] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615 [term 1 FOLLOWER]: Refusing update from remote peer b5e31e64795d48fd98784565b108a43a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:45.329151 10304 raft_consensus.cc:1261] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452 [term 1 FOLLOWER]: Refusing update from remote peer b5e31e64795d48fd98784565b108a43a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:16:45.329942 10507 consensus_queue.cc:1035] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f68ccd937954663b0e9637e1aa57615" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:45.330204 10512 consensus_queue.cc:1035] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a6608fef39148209bdf7f7f8caad452" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 45103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:16:45.338724 10517 mvcc.cc:205] Tried to move back new op lower bound from 7333698786620080128 to 7333698786394255360. Current Snapshot: MvccSnapshot[applied={T|T < 7333698786620080128}]
I20260926 20:16:45.347059 10025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:45.347223 10025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:16:45.350039 10025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fafcbe7518dd47eda6dda5af6f9f3771: Sending DeleteTablet for 3 replicas of tablet 12cbd0fc8d9b43128ce1565b81364c10
I20260926 20:16:45.351007 10150 tablet_service.cc:1514] Processing DeleteTablet for tablet 12cbd0fc8d9b43128ce1565b81364c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:45 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:53966
I20260926 20:16:45.351189 10284 tablet_service.cc:1514] Processing DeleteTablet for tablet 12cbd0fc8d9b43128ce1565b81364c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:45 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:45092
I20260926 20:16:45.351608 10524 tablet_replica.cc:330] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a: stopping tablet replica
I20260926 20:16:45.351872 10418 tablet_service.cc:1514] Processing DeleteTablet for tablet 12cbd0fc8d9b43128ce1565b81364c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:16:45 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:40140
I20260926 20:16:45.351990 10524 raft_consensus.cc:2229] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:16:45.352402 10524 raft_consensus.cc:2258] T 12cbd0fc8d9b43128ce1565b81364c10 P b5e31e64795d48fd98784565b108a43a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:45.352957 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 10087
I20260926 20:16:45.355145 10525 tablet_replica.cc:330] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: stopping tablet replica
I20260926 20:16:45.355523 10525 raft_consensus.cc:2229] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:45.355899 10525 raft_consensus.cc:2258] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:45.356287 10526 tablet_replica.cc:330] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: stopping tablet replica
I20260926 20:16:45.356640 10526 raft_consensus.cc:2229] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:16:45.356981 10526 raft_consensus.cc:2258] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:16:45.357739 10525 ts_tablet_manager.cc:1905] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:45.358646 10526 ts_tablet_manager.cc:1905] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:16:45.360127 10525 ts_tablet_manager.cc:1918] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:16:45.360242 10525 log.cc:1196] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/12cbd0fc8d9b43128ce1565b81364c10
I20260926 20:16:45.360553 10525 ts_tablet_manager.cc:1939] T 12cbd0fc8d9b43128ce1565b81364c10 P 3a6608fef39148209bdf7f7f8caad452: Deleting consensus metadata
I20260926 20:16:45.361889 10526 ts_tablet_manager.cc:1918] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 20:16:45.361889 10010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:43583 (error 108)
I20260926 20:16:45.362000 10526 log.cc:1196] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/12cbd0fc8d9b43128ce1565b81364c10
I20260926 20:16:45.362339 10526 ts_tablet_manager.cc:1939] T 12cbd0fc8d9b43128ce1565b81364c10 P 3f68ccd937954663b0e9637e1aa57615: Deleting consensus metadata
I20260926 20:16:45.362452 10010 catalog_manager.cc:4841] TS 3a6608fef39148209bdf7f7f8caad452 (127.31.119.194:45103): tablet 12cbd0fc8d9b43128ce1565b81364c10 (table test-table [id=e94e76afdbf441c794dd088a5bac12eb]) successfully deleted
W20260926 20:16:45.363086 10499 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:16:45.362792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:43583 (local address 127.0.0.1:53984)
0926 20:16:45.362944 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:16:45.363007 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:43583: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 20:16:45.363252 10010 catalog_manager.cc:4584] TS b5e31e64795d48fd98784565b108a43a (127.31.119.193:43583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12cbd0fc8d9b43128ce1565b81364c10: Network error: Client connection negotiation failed: client connection to 127.31.119.193:43583: connect: Connection reset by peer (error 104)
I20260926 20:16:45.363554 10010 catalog_manager.cc:4841] TS 3f68ccd937954663b0e9637e1aa57615 (127.31.119.195:41503): tablet 12cbd0fc8d9b43128ce1565b81364c10 (table test-table [id=e94e76afdbf441c794dd088a5bac12eb]) successfully deleted
I20260926 20:16:45.363570 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 10221
I20260926 20:16:45.370487 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 10355
I20260926 20:16:45.376246 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 9993
2026-09-26T20:16:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2447 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:16:45 dist-test-slave-vnbh krb5kdc[10530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:16:45 dist-test-slave-vnbh krb5kdc[10530](info): set up 2 sockets
Sep 26 20:16:45 dist-test-slave-vnbh krb5kdc[10530](info): commencing operation
krb5kdc: starting...
W20260926 20:16:46.411616 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:16:46 dist-test-slave-vnbh krb5kdc[10530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:16:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.119.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.119.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/rangeradmin_127.31.119.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/rangeradmin_127.31.119.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.119.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.119.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/rangerlookup_127.31.119.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/rangerlookup_127.31.119.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.119.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.119.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/HTTP_127.31.119.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/HTTP_127.31.119.212@KRBTEST.COM.keytab.
I20260926 20:16:46.592635 32223 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYRmynJ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/postgres -l logfile start
2026-09-26 20:16:49.261 UTC [10560] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:16:49.261 UTC [10560] LOG: listening on IPv4 address "127.31.119.212", port 49253
2026-09-26 20:16:49.266 UTC [10560] LOG: listening on Unix socket "/tmp/.s.PGSQL.49253"
2026-09-26 20:16:49.301 UTC [10563] LOG: database system was shut down at 2026-09-26 20:16:47 UTC
2026-09-26 20:16:49.306 UTC [10560] LOG: database system is ready to accept connections
I20260926 20:16:50.271147 32223 mini_postgres.cc:96] Postgres bound to 49253
2026-09-26 20:16:50.277 UTC [10571] FATAL: database "slave" does not exist
127.31.119.212:49253 - accepting connections
I20260926 20:16:50.278403 32223 mini_ranger.cc:161] Starting Ranger...
I20260926 20:16:50.294291 32223 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:16:50.863695 32223 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:16:50.863795 32223 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-admin
2026-09-26 20:16:51,148 [I] DB FLAVOR :POSTGRES
2026-09-26 20:16:51,148 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:16:51,149 [I] Checking connection..
2026-09-26 20:16:51,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:16:51,429 [I] Checking connection passed.
2026-09-26 20:16:51,429 [I] --------- Verifying version history table ---------
2026-09-26 20:16:51,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:16:51,705 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:16:51,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:16:51,961 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:16:51,962 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:16:51,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:16:52.221 UTC [10635] WARNING: there is no transaction in progress
2026-09-26 20:16:52,228 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:16:52,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:16:52,492 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:16:52,492 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:16:52,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:16:52,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:16:52,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vnbh.c.gcp-upstream.internal','N') ;"
2026-09-26 20:16:53,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:16:53,461 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:16:53,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:16:53,735 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:16:53,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:16:53,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:16:54,248 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:16:54,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:16:55.292 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.309 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.328 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.346 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.365 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.389 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.459 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.467 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.484 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.496 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.508 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.525 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.536 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.543 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.552 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:55.560 UTC [10743] WARNING: there is no transaction in progress
2026-09-26 20:16:56,372 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:16:56,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:16:56,629 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:16:56,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:16:56,898 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:16:56,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:16:57,171 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:16:57,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:16:57,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:16:57,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vnbh.c.gcp-upstream.internal';"
2026-09-26 20:16:57,923 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:16:57,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:16:58,170 [I] DB_PATCHES have already been applied
I20260926 20:16:58.176970 32223 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYRmynJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:16:58.177038 32223 mini_ranger.cc:193] Using host: 127.31.119.212
I20260926 20:16:58.180881 32223 mini_ranger.cc:227] Ranger admin URL: http://127.31.119.212:33531
Sep 26, 2026 8:16:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:16:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:16:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:16:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:16:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:16:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:16:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:16:59.306723 32223 mini_ranger.cc:160] Time spent starting Ranger: real 9.028s user 0.000s sys 0.011s
Sep 26, 2026 8:17:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:17:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 8:17:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:17:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:17:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:17:20.730077 32223 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.119.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.119.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/rangerkms_127.31.119.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/rangerkms_127.31.119.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.119.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.119.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.119.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/HTTP_127.31.119.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.119.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/HTTP_127.31.119.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:17:20 dist-test-slave-vnbh krb5kdc[10530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453840, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:17:20.819015 32223 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:17:20.829514 32223 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:17:21.785841 32223 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:17:21.785923 32223 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-kms
I20260926 20:17:21.786015 32223 mini_ranger_kms.cc:227] Using postgres at 127.31.119.212:49253
2026-09-26 20:17:21,961 [I] DB FLAVOR :POSTGRES
2026-09-26 20:17:21,961 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:17:21,962 [I] Checking connection
2026-09-26 20:17:21,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:17:22,199 [I] connection success
2026-09-26 20:17:22,199 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:17:22,199 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:17:22,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:17:22,464 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:17:22,464 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:17:22,464 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:17:22,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:49253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:17:22,746 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:17:23.258622 32223 mini_ranger_kms.cc:310] Created kms service
I20260926 20:17:23.453424 32223 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:17:23.496346 32223 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:17:23 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453840, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:17:23.928449 32223 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:17:23.928617 32223 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:17:23.928655 32223 mini_ranger_kms.cc:240] Using host: 127.31.119.212
I20260926 20:17:23.932420 32223 mini_ranger_kms.cc:276] Ranger KMS PID: 10923
I20260926 20:17:23.932587 32223 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.119.212:43563
Sep 26, 2026 8:17:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:17:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:17:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:17:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:17:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:17:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 20:17:26.190697 32223 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.372s user 0.001s sys 0.160s
I20260926 20:17:26.190851 32223 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:17:26.190990 32223 mini_ranger_kms.cc:401] 127.31.119.212:43563/kms/v1/keys
Sep 26 20:17:26 dist-test-slave-vnbh krb5kdc[10530](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790453846, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.119.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:17:26 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453846, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.119.212@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
Sep 26 20:17:27 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453846, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.119.212@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
Sep 26 20:17:29 dist-test-slave-vnbh krb5kdc[10530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:17:29.712952 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:45995
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:45995
--ranger_config_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:43563/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:17:29.838060 10974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:17:29.838425 10974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:17:29.838515 10974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:17:29.842849 10974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:17:29.842952 10974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:17:29.842989 10974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:17:29.843045 10974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:17:29.843098 10974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:17:29.849063 10974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.119.212:43563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:45995
--ranger_config_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:45995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:17:29.850318 10974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:17:29.851432 10974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:17:29.861507 10979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:17:29.861532 10980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:17:29.861579 10974 server_base.cc:1034] running on GCE node
W20260926 20:17:29.861519 10982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:17:29.862509 10974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:17:29.863854 10974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:17:29.865051 10974 hybrid_clock.cc:648] HybridClock initialized: now 1790453849865021 us; error 60 us; skew 500 ppm
Sep 26 20:17:29 dist-test-slave-vnbh krb5kdc[10530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:17:29.875228 10974 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:17:29.876394 10974 webserver.cc:460] Webserver started at http://127.31.119.254:34807/ using document root <none> and password file <none>
I20260926 20:17:29.877020 10974 fs_manager.cc:359] Metadata directory not provided
I20260926 20:17:29.877107 10974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:17:29.877393 10974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:17:29 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:17:29.938254 10974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "0795dd2fb85a454f83a47f5460f5799a"
format_stamp: "Formatted at 2026-09-26 20:17:29 on dist-test-slave-vnbh"
server_key: "9f51d3b91b671b0e3e5e56d19f0381b8"
server_key_iv: "6805412ebcd98df3e4bc752b77555240"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:29.939064 10974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "0795dd2fb85a454f83a47f5460f5799a"
format_stamp: "Formatted at 2026-09-26 20:17:29 on dist-test-slave-vnbh"
server_key: "9f51d3b91b671b0e3e5e56d19f0381b8"
server_key_iv: "6805412ebcd98df3e4bc752b77555240"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:29.944360 10974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:17:29.962899 10989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:17:29.964350 10974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:17:29.964506 10974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "0795dd2fb85a454f83a47f5460f5799a"
format_stamp: "Formatted at 2026-09-26 20:17:29 on dist-test-slave-vnbh"
server_key: "9f51d3b91b671b0e3e5e56d19f0381b8"
server_key_iv: "6805412ebcd98df3e4bc752b77555240"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:29.964651 10974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:17:29.982805 10974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:17:29.986066 10974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:17:29.986285 10974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:17:29.994340 10974 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:45995
I20260926 20:17:29.994386 11041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:45995 every 8 connection(s)
I20260926 20:17:29.995400 10974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:17:29.998610 11042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:17:30.002918 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 10974
I20260926 20:17:30.003062 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:17:30.003363 32223 external_mini_cluster.cc:1428] Setting key b57bf993314d312414747cfbb529ab92
I20260926 20:17:30.005196 11042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a: Bootstrap starting.
I20260926 20:17:30.008090 11042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a: Neither blocks nor log segments found. Creating new log.
I20260926 20:17:30.009073 11042 log.cc:824] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:17:30.011508 11042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a: No bootstrap required, opened a new log
Sep 26 20:17:30 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:17:30.016532 11042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0795dd2fb85a454f83a47f5460f5799a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45995 } }
I20260926 20:17:30.016914 11042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:17:30.017014 11042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0795dd2fb85a454f83a47f5460f5799a, State: Initialized, Role: FOLLOWER
I20260926 20:17:30.017643 11042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0795dd2fb85a454f83a47f5460f5799a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45995 } }
I20260926 20:17:30.017933 11042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:17:30.018030 11042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:17:30.018153 11042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:17:30.019429 11042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0795dd2fb85a454f83a47f5460f5799a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45995 } }
I20260926 20:17:30.019891 11042 leader_election.cc:302] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0795dd2fb85a454f83a47f5460f5799a; no voters:
I20260926 20:17:30.020309 11042 leader_election.cc:288] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:17:30.020705 11048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:17:30.021520 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:30.005050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.31.119.254:45995)
0926 20:17:30.005575 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:30.005594 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:30.005638 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:17:30.006534 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:30.006559 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:30.007006 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:30.007404 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:17:30.008340 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:30.009804 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:30.011017 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:30.011768 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:30.011942 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:30.015353 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:17:30.015361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:17:30.015384 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:17:30.015425 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:17:30.017680 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:17:30.018276 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:17:30.018283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:17:30.018290 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:17:30.018439 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:17:30.018799 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:17:30.018802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:17:30.018804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:17:30.019408 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:17:30.019683 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:30.019996 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 20:17:30.020360 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":181,"threads_started":1}
I20260926 20:17:30.021610 11048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [term 1 LEADER]: Becoming Leader. State: Replica: 0795dd2fb85a454f83a47f5460f5799a, State: Running, Role: LEADER
I20260926 20:17:30.021714 11042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:17:30.021955 11048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0795dd2fb85a454f83a47f5460f5799a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45995 } }
I20260926 20:17:30.023375 11048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0795dd2fb85a454f83a47f5460f5799a. Latest consensus state: current_term: 1 leader_uuid: "0795dd2fb85a454f83a47f5460f5799a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0795dd2fb85a454f83a47f5460f5799a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45995 } } }
I20260926 20:17:30.023506 11048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [sys.catalog]: This master's current role is: LEADER
I20260926 20:17:30.023991 11042 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:17:30.024509 11047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0795dd2fb85a454f83a47f5460f5799a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0795dd2fb85a454f83a47f5460f5799a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45995 } } }
I20260926 20:17:30.024682 11047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a [sys.catalog]: This master's current role is: LEADER
I20260926 20:17:30.027578 11061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:17:30.031991 11061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:17:30.039088 11061 catalog_manager.cc:1347] Generated new cluster ID: 962f72f9cdbe4becb6ae0cf01d12bbda
I20260926 20:17:30.039165 11061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:17:30.048027 11061 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:17:30.049499 11061 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:17:30.059616 11061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a: Generated new TSK 0
I20260926 20:17:30.060572 11061 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:17:30.824249 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:43563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:45995
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:17:30.948361 11087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:17:30.948696 11087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:17:30.948786 11087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:17:30.953390 11087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:17:30.953481 11087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:17:30.953603 11087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:17:30.959348 11087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.119.212:43563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:45995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:17:30.960605 11087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:17:30.961831 11087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:17:30.972529 11093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:17:30.972638 11087 server_base.cc:1034] running on GCE node
W20260926 20:17:30.972529 11092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:17:30.972853 11095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:17:30.973337 11087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:17:30.974110 11087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:17:30.975308 11087 hybrid_clock.cc:648] HybridClock initialized: now 1790453850975288 us; error 35 us; skew 500 ppm
Sep 26 20:17:30 dist-test-slave-vnbh krb5kdc[10530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:17:30.978515 11087 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:17:30.979594 11087 webserver.cc:460] Webserver started at http://127.31.119.193:41343/ using document root <none> and password file <none>
I20260926 20:17:30.980234 11087 fs_manager.cc:359] Metadata directory not provided
I20260926 20:17:30.980330 11087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:17:30.980574 11087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:17:30 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:17:31.007200 11087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "eee526e1dc984352b0a297d99d6475a9"
format_stamp: "Formatted at 2026-09-26 20:17:31 on dist-test-slave-vnbh"
server_key: "98ff5a5293d2ee1b1dfaa58349daf67f"
server_key_iv: "65d6d80ef5490468e05130d23fcacdd1"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:31.008059 11087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "eee526e1dc984352b0a297d99d6475a9"
format_stamp: "Formatted at 2026-09-26 20:17:31 on dist-test-slave-vnbh"
server_key: "98ff5a5293d2ee1b1dfaa58349daf67f"
server_key_iv: "65d6d80ef5490468e05130d23fcacdd1"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:31.013649 11087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 20:17:31.028673 11104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:17:31.030471 11087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:17:31.030610 11087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "eee526e1dc984352b0a297d99d6475a9"
format_stamp: "Formatted at 2026-09-26 20:17:31 on dist-test-slave-vnbh"
server_key: "98ff5a5293d2ee1b1dfaa58349daf67f"
server_key_iv: "65d6d80ef5490468e05130d23fcacdd1"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:31.030750 11087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:17:31.063493 11087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:17:31.067123 11087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:17:31.067354 11087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:17:31.068022 11087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:17:31.068995 11087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:17:31.069072 11087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:17:31.069211 11087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:17:31.069260 11087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:17:31.080005 11087 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:39975
I20260926 20:17:31.080245 11217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:39975 every 8 connection(s)
I20260926 20:17:31.081095 11087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:17:31.083879 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 11087
I20260926 20:17:31.084023 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:17:31.084350 32223 external_mini_cluster.cc:1428] Setting key b2d57078b9f8c43137d08fa963f0dc55
Sep 26 20:17:31 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:17:31.099967 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.083232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:52857 (local address 127.31.119.254:45995)
0926 20:17:31.083528 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:31.083532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:31.084376 (+ 844us) server_negotiation.cc:407] Connection header received
0926 20:17:31.085712 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:31.085715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:31.085768 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:31.085832 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:17:31.087172 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.087725 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.089645 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.090011 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.090090 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.095147 (+ 5057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:17:31.095151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:17:31.095154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:17:31.095186 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:17:31.097198 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:17:31.097997 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:17:31.098002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:17:31.098004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:17:31.098102 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:17:31.098518 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:17:31.098522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:17:31.098524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:17:31.098769 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:17:31.098852 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:31.099620 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 20:17:31.099773 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260926 20:17:31.102032 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.083712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:45995 (local address 127.31.119.193:52857)
0926 20:17:31.084210 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 20:17:31.084267 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:17:31.085467 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:17:31.086235 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:17:31.086242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:17:31.086516 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:17:31.087015 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.087029 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.087913 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.087918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.089499 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.089510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.090313 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.090317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.090527 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.092320 (+ 1793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:17:31.092345 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:17:31.094896 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:17:31.097406 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:17:31.097413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:17:31.097434 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:17:31.097877 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:17:31.098221 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:17:31.098224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:17:31.098226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:17:31.098409 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:17:31.098918 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:17:31.098925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:17:31.099327 (+ 402us) client_negotiation.cc:769] Sending connection context
0926 20:17:31.100500 (+ 1173us) client_negotiation.cc:240] Negotiation successful
0926 20:17:31.100882 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":138,"threads_started":1}
I20260926 20:17:31.103809 11218 heartbeater.cc:347] Connected to a master server at 127.31.119.254:45995
I20260926 20:17:31.104069 11218 heartbeater.cc:464] Registering TS with master...
I20260926 20:17:31.104671 11218 heartbeater.cc:511] Master 127.31.119.254:45995 requested a full tablet report, sending...
I20260926 20:17:31.106599 11006 ts_manager.cc:194] Registered new tserver with Master: eee526e1dc984352b0a297d99d6475a9 (127.31.119.193:39975)
I20260926 20:17:31.108168 11006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:52857
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:17:31.144963 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:43563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:45995
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:17:31 dist-test-slave-vnbh krb5kdc[10530](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790453851, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:17:31.318140 11226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:17:31.318558 11226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:17:31.318625 11226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:17:31.324565 11226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:17:31.324667 11226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:17:31.324761 11226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:17:31.331988 11226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.119.212:43563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:45995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:17:31.333704 11226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:17:31.335119 11226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:17:31.347936 11233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:17:31.347961 11235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:17:31.347967 11232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:17:31.348579 11226 server_base.cc:1034] running on GCE node
I20260926 20:17:31.349089 11226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:17:31.349994 11226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:17:31.351186 11226 hybrid_clock.cc:648] HybridClock initialized: now 1790453851351131 us; error 75 us; skew 500 ppm
Sep 26 20:17:31 dist-test-slave-vnbh krb5kdc[10530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453851, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:17:31.355016 11226 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:17:31.356235 11226 webserver.cc:460] Webserver started at http://127.31.119.194:33115/ using document root <none> and password file <none>
I20260926 20:17:31.357175 11226 fs_manager.cc:359] Metadata directory not provided
I20260926 20:17:31.357352 11226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:17:31.357645 11226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:17:31 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453851, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:17:31.375185 11226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "170263bb913b4460a501597eb3b1cfdb"
format_stamp: "Formatted at 2026-09-26 20:17:31 on dist-test-slave-vnbh"
server_key: "6f0e0cd0915ff92429011b4c4f86b164"
server_key_iv: "b2439637d8253cd92ec4b94ae89b3eea"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:31.376120 11226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "170263bb913b4460a501597eb3b1cfdb"
format_stamp: "Formatted at 2026-09-26 20:17:31 on dist-test-slave-vnbh"
server_key: "6f0e0cd0915ff92429011b4c4f86b164"
server_key_iv: "b2439637d8253cd92ec4b94ae89b3eea"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:31.390272 11226 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.007s sys 0.000s
I20260926 20:17:31.414186 11242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:17:31.416062 11226 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:17:31.416209 11226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "170263bb913b4460a501597eb3b1cfdb"
format_stamp: "Formatted at 2026-09-26 20:17:31 on dist-test-slave-vnbh"
server_key: "6f0e0cd0915ff92429011b4c4f86b164"
server_key_iv: "b2439637d8253cd92ec4b94ae89b3eea"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:31.416380 11226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:17:31.441740 11226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:17:31.445967 11226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:17:31.446337 11226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:17:31.447149 11226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:17:31.448338 11226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:17:31.448457 11226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:17:31.448565 11226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:17:31.448638 11226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:17:31.461822 11226 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:46593
I20260926 20:17:31.463274 11226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:17:31.464246 11355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:46593 every 8 connection(s)
I20260926 20:17:31.466447 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 11226
I20260926 20:17:31.466572 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:17:31.466874 32223 external_mini_cluster.cc:1428] Setting key 452426fabb75d30e032b316665ac9b4e
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
Sep 26 20:17:31 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453851, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:17:31.503602 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.468955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:38347 (local address 127.31.119.254:45995)
0926 20:17:31.469141 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:31.469147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:31.470647 (+ 1500us) server_negotiation.cc:407] Connection header received
0926 20:17:31.472328 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:31.472332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:31.472472 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:31.472612 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:17:31.490302 (+ 17690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.491230 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.492812 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.493307 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.493427 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.497076 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:17:31.497083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:17:31.497089 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:17:31.497125 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:17:31.499822 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:17:31.500649 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:17:31.500656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:17:31.500661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:17:31.500771 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:17:31.501403 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:17:31.501408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:17:31.501410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:17:31.501681 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:17:31.501763 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:31.503255 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0926 20:17:31.503422 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 20:17:31.504839 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.469872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:45995 (local address 127.31.119.194:38347)
0926 20:17:31.470401 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 20:17:31.470548 (+ 147us) client_negotiation.cc:174] Beginning negotiation
0926 20:17:31.471992 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:17:31.473514 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:17:31.473529 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:17:31.473951 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:17:31.490045 (+ 16094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.490084 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.491397 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.491403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.492663 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.492678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.493482 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.493486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.493729 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.494440 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:17:31.494465 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:17:31.496832 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:17:31.499979 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:17:31.499987 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:17:31.500010 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:17:31.500487 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:17:31.500912 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:17:31.500916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:17:31.500920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:17:31.501194 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:17:31.502430 (+ 1236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:17:31.502442 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:17:31.502884 (+ 442us) client_negotiation.cc:769] Sending connection context
0926 20:17:31.503115 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 20:17:31.503523 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":159,"threads_started":1}
I20260926 20:17:31.506323 11356 heartbeater.cc:347] Connected to a master server at 127.31.119.254:45995
I20260926 20:17:31.506636 11356 heartbeater.cc:464] Registering TS with master...
I20260926 20:17:31.507344 11356 heartbeater.cc:511] Master 127.31.119.254:45995 requested a full tablet report, sending...
I20260926 20:17:31.508751 11006 ts_manager.cc:194] Registered new tserver with Master: 170263bb913b4460a501597eb3b1cfdb (127.31.119.194:46593)
I20260926 20:17:31.509615 11006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:38347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:17:31.520826 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:43563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:45995
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:17:31.668884 11361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:17:31.669210 11361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:17:31.669358 11361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:17:31.673784 11361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:17:31.673877 11361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:17:31.673985 11361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:17:31.679620 11361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.119.212:43563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:45995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:17:31.680871 11361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:17:31.682147 11361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:17:31.693606 11367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:17:31.693904 11369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:17:31.693606 11366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:17:31.694284 11361 server_base.cc:1034] running on GCE node
I20260926 20:17:31.694800 11361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:17:31.695575 11361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:17:31.697752 11361 hybrid_clock.cc:648] HybridClock initialized: now 1790453851697714 us; error 54 us; skew 500 ppm
Sep 26 20:17:31 dist-test-slave-vnbh krb5kdc[10530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453851, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:17:31.701933 11361 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:17:31.703243 11361 webserver.cc:460] Webserver started at http://127.31.119.195:34891/ using document root <none> and password file <none>
I20260926 20:17:31.704097 11361 fs_manager.cc:359] Metadata directory not provided
I20260926 20:17:31.704257 11361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:17:31.704564 11361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:17:31 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453851, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:17:31.724696 11361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "499ca67570af49e08f56da495b9e9675"
format_stamp: "Formatted at 2026-09-26 20:17:31 on dist-test-slave-vnbh"
server_key: "77cf1004f2ef31a079b95fff663d4ea5"
server_key_iv: "8854974034c0fd05baa232633d65b06c"
server_key_version: "kuduclusterkey@0"
Sep 26 20:17:31 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453851, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:17:31.730413 11361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "499ca67570af49e08f56da495b9e9675"
format_stamp: "Formatted at 2026-09-26 20:17:31 on dist-test-slave-vnbh"
server_key: "77cf1004f2ef31a079b95fff663d4ea5"
server_key_iv: "8854974034c0fd05baa232633d65b06c"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:31.743168 11361 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.001s
I20260926 20:17:31.779486 11376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:17:31.781507 11361 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.002s
I20260926 20:17:31.781752 11361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "499ca67570af49e08f56da495b9e9675"
format_stamp: "Formatted at 2026-09-26 20:17:31 on dist-test-slave-vnbh"
server_key: "77cf1004f2ef31a079b95fff663d4ea5"
server_key_iv: "8854974034c0fd05baa232633d65b06c"
server_key_version: "kuduclusterkey@0"
I20260926 20:17:31.782006 11361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:17:31.800127 11361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:17:31.804117 11361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:17:31.804428 11361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:17:31.805253 11361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:17:31.806548 11361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:17:31.806654 11361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:17:31.806751 11361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:17:31.806813 11361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:17:31.820256 11361 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:40627
I20260926 20:17:31.820274 11489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:40627 every 8 connection(s)
I20260926 20:17:31.821583 11361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:17:31.826467 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 11361
I20260926 20:17:31.826603 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:17:31.826911 32223 external_mini_cluster.cc:1428] Setting key 5de53a2ed8c51b8a539375d54c17648f
Sep 26 20:17:31 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453851, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:17:31.837769 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.824188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:55193 (local address 127.31.119.254:45995)
0926 20:17:31.824350 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:31.824355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:31.825376 (+ 1021us) server_negotiation.cc:407] Connection header received
0926 20:17:31.826508 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:31.826512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:31.826579 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:31.826642 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:17:31.828855 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.829588 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.830816 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.831065 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.831181 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.833719 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:17:31.833722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:17:31.833724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:17:31.833749 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:17:31.835248 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:17:31.835904 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:17:31.835908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:17:31.835910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:17:31.836041 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:17:31.836425 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:17:31.836428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:17:31.836430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:17:31.836718 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:17:31.836828 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:31.837518 (+ 690us) server_negotiation.cc:299] Negotiation successful
0926 20:17:31.837643 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:17:31.839193 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.824652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:45995 (local address 127.31.119.195:55193)
0926 20:17:31.825182 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 20:17:31.825239 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:17:31.826222 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:17:31.827106 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:17:31.827116 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:17:31.827685 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:17:31.828440 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.828463 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.829713 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.829722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.830692 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.830701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.831220 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.831225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.831401 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.832059 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:17:31.832076 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:17:31.833542 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:17:31.835388 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:17:31.835394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:17:31.835412 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:17:31.835791 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:17:31.836149 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:17:31.836152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:17:31.836154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:17:31.836314 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:17:31.836825 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:17:31.836833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:17:31.837115 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 20:17:31.838042 (+ 927us) client_negotiation.cc:240] Negotiation successful
0926 20:17:31.838348 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":168,"threads_started":1}
I20260926 20:17:31.840634 11490 heartbeater.cc:347] Connected to a master server at 127.31.119.254:45995
I20260926 20:17:31.840878 11490 heartbeater.cc:464] Registering TS with master...
I20260926 20:17:31.841530 11490 heartbeater.cc:511] Master 127.31.119.254:45995 requested a full tablet report, sending...
I20260926 20:17:31.842581 11005 ts_manager.cc:194] Registered new tserver with Master: 499ca67570af49e08f56da495b9e9675 (127.31.119.195:40627)
I20260926 20:17:31.843173 11005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:55193
I20260926 20:17:31.853755 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:17:31.866077 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.856290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.31.119.254:45995)
0926 20:17:31.856411 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:31.856415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:31.857122 (+ 707us) server_negotiation.cc:407] Connection header received
0926 20:17:31.857370 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:31.857374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:31.857425 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:31.857480 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:17:31.858547 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.859276 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.860650 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.860949 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.861031 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.862697 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:17:31.862701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:17:31.862703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:17:31.862735 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:17:31.864267 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:17:31.864713 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:17:31.864718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:17:31.864720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:17:31.864802 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:17:31.865109 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:17:31.865112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:17:31.865114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:17:31.865370 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:17:31.865505 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:31.865767 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 20:17:31.865879 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:17:31.869571 11005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:17:31.872730 11005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:17:31.895457 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.884853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:46593 (local address 127.0.0.1:48942)
0926 20:17:31.886669 (+ 1816us) negotiation.cc:107] Waiting for socket to connect
0926 20:17:31.886729 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:17:31.886903 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:17:31.887780 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:17:31.887787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:17:31.887811 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:17:31.888035 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.888044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.890289 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.890292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.892432 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.892443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.894002 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.894006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.894125 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.894167 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:17:31.895159 (+ 992us) client_negotiation.cc:240] Negotiation successful
0926 20:17:31.895295 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":39,"thread_start_us":140,"threads_started":1}
I20260926 20:17:31.896097 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.884998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.31.119.194:46593)
0926 20:17:31.885421 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:31.885434 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:31.886828 (+ 1394us) server_negotiation.cc:407] Connection header received
0926 20:17:31.887130 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:31.887137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:31.887368 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:31.887529 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:17:31.889028 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.890152 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.892616 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.893663 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.893785 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.893934 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:31.895832 (+ 1898us) server_negotiation.cc:299] Negotiation successful
0926 20:17:31.895950 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":112,"threads_started":1}
I20260926 20:17:31.896636 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.885553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:40627 (local address 127.0.0.1:45142)
0926 20:17:31.886669 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0926 20:17:31.886728 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:17:31.886852 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:17:31.887806 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:17:31.887809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:17:31.887825 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:17:31.888158 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.888166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.890672 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.890675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.891977 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.891987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.896243 (+ 4256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.896247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.896355 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.896378 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:17:31.896444 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:17:31.896508 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":203,"thread_start_us":88,"threads_started":1}
I20260926 20:17:31.897375 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.885717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.31.119.195:40627)
0926 20:17:31.887234 (+ 1517us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:31.887245 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:31.887273 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:17:31.887342 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:31.887348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:31.887533 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:31.887671 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:17:31.888284 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.890550 (+ 2266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.895062 (+ 4512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.895920 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.896051 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.896181 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:31.897105 (+ 924us) server_negotiation.cc:299] Negotiation successful
0926 20:17:31.897215 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1416,"thread_start_us":90,"threads_started":1}
I20260926 20:17:31.899225 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.886129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:39975 (local address 127.0.0.1:43864)
0926 20:17:31.889499 (+ 3370us) negotiation.cc:107] Waiting for socket to connect
0926 20:17:31.889513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:17:31.889599 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:17:31.892101 (+ 2502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:17:31.892104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:17:31.892122 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:17:31.892198 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.892202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.893704 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.893708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.894884 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:31.894894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.898780 (+ 3886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.898784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:31.898879 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.898902 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:17:31.899008 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 20:17:31.899088 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3309,"thread_start_us":83,"threads_started":1}
I20260926 20:17:31.900751 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:31.886274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.31.119.193:39975)
0926 20:17:31.889399 (+ 3125us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:31.889406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:31.889685 (+ 279us) server_negotiation.cc:407] Connection header received
0926 20:17:31.889761 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:31.889766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:31.889972 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:31.890098 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:17:31.892315 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.893574 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.897608 (+ 4034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:31.898471 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:31.898595 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:31.898716 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:31.900494 (+ 1778us) server_negotiation.cc:299] Negotiation successful
0926 20:17:31.900613 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2993,"thread_start_us":105,"threads_started":1}
I20260926 20:17:31.900794 11283 tablet_service.cc:1467] Processing CreateTablet for tablet 98aa2e865fc6475a9a195101faf0492e (DEFAULT_TABLE table=test-table [id=5f55d8cababc48f584c0bc188e4abd55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:17:31.902037 11283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98aa2e865fc6475a9a195101faf0492e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:17:31.903856 11151 tablet_service.cc:1467] Processing CreateTablet for tablet 98aa2e865fc6475a9a195101faf0492e (DEFAULT_TABLE table=test-table [id=5f55d8cababc48f584c0bc188e4abd55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:17:31.904987 11151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98aa2e865fc6475a9a195101faf0492e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:17:31.907356 11424 tablet_service.cc:1467] Processing CreateTablet for tablet 98aa2e865fc6475a9a195101faf0492e (DEFAULT_TABLE table=test-table [id=5f55d8cababc48f584c0bc188e4abd55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:17:31.908829 11424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98aa2e865fc6475a9a195101faf0492e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:17:31.913839 11508 tablet_bootstrap.cc:492] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9: Bootstrap starting.
I20260926 20:17:31.921442 11510 tablet_bootstrap.cc:492] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: Bootstrap starting.
I20260926 20:17:31.921902 11509 tablet_bootstrap.cc:492] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: Bootstrap starting.
I20260926 20:17:31.925136 11509 tablet_bootstrap.cc:654] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: Neither blocks nor log segments found. Creating new log.
I20260926 20:17:31.925136 11510 tablet_bootstrap.cc:654] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: Neither blocks nor log segments found. Creating new log.
I20260926 20:17:31.926137 11510 log.cc:824] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:17:31.926137 11509 log.cc:824] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: Log is configured to *not* fsync() on all Append() calls
I20260926 20:17:31.927419 11508 tablet_bootstrap.cc:654] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9: Neither blocks nor log segments found. Creating new log.
I20260926 20:17:31.928368 11508 log.cc:824] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:17:31.930785 11508 tablet_bootstrap.cc:492] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9: No bootstrap required, opened a new log
I20260926 20:17:31.930996 11508 ts_tablet_manager.cc:1397] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20260926 20:17:31.931910 11509 tablet_bootstrap.cc:492] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: No bootstrap required, opened a new log
I20260926 20:17:31.932096 11509 ts_tablet_manager.cc:1397] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260926 20:17:31.936249 11508 raft_consensus.cc:348] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 } } peers { permanent_uuid: "499ca67570af49e08f56da495b9e9675" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40627 } } peers { permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 } }
I20260926 20:17:31.936717 11508 raft_consensus.cc:374] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:17:31.936856 11508 raft_consensus.cc:729] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee526e1dc984352b0a297d99d6475a9, State: Initialized, Role: FOLLOWER
I20260926 20:17:31.938521 11509 raft_consensus.cc:348] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 } } peers { permanent_uuid: "499ca67570af49e08f56da495b9e9675" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40627 } } peers { permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 } }
I20260926 20:17:31.938813 11509 raft_consensus.cc:374] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:17:31.938858 11509 raft_consensus.cc:729] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499ca67570af49e08f56da495b9e9675, State: Initialized, Role: FOLLOWER
I20260926 20:17:31.939141 11510 tablet_bootstrap.cc:492] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: No bootstrap required, opened a new log
I20260926 20:17:31.939087 11508 consensus_queue.cc:260] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 } } peers { permanent_uuid: "499ca67570af49e08f56da495b9e9675" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40627 } } peers { permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 } }
I20260926 20:17:31.939379 11510 ts_tablet_manager.cc:1397] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260926 20:17:31.940461 11508 ts_tablet_manager.cc:1428] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
I20260926 20:17:31.940922 11218 heartbeater.cc:503] Master 127.31.119.254:45995 was elected leader, sending a full tablet report...
I20260926 20:17:31.941677 11509 consensus_queue.cc:260] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 } } peers { permanent_uuid: "499ca67570af49e08f56da495b9e9675" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40627 } } peers { permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 } }
I20260926 20:17:31.942934 11509 ts_tablet_manager.cc:1428] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260926 20:17:31.943349 11490 heartbeater.cc:503] Master 127.31.119.254:45995 was elected leader, sending a full tablet report...
I20260926 20:17:31.943435 11510 raft_consensus.cc:348] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 } } peers { permanent_uuid: "499ca67570af49e08f56da495b9e9675" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40627 } } peers { permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 } }
I20260926 20:17:31.943719 11510 raft_consensus.cc:374] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:17:31.943771 11510 raft_consensus.cc:729] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170263bb913b4460a501597eb3b1cfdb, State: Initialized, Role: FOLLOWER
I20260926 20:17:31.944291 11510 consensus_queue.cc:260] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 } } peers { permanent_uuid: "499ca67570af49e08f56da495b9e9675" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40627 } } peers { permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 } }
I20260926 20:17:31.945359 11510 ts_tablet_manager.cc:1428] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:17:31.945669 11356 heartbeater.cc:503] Master 127.31.119.254:45995 was elected leader, sending a full tablet report...
W20260926 20:17:31.965497 11357 tablet.cc:2369] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:17:32 dist-test-slave-vnbh krb5kdc[10530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453851, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
W20260926 20:17:32.073132 11491 tablet.cc:2369] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:17:32.099318 11219 tablet.cc:2369] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:17:32.243347 11515 raft_consensus.cc:482] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:17:32.243634 11515 raft_consensus.cc:504] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 } } peers { permanent_uuid: "499ca67570af49e08f56da495b9e9675" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40627 } } peers { permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 } }
I20260926 20:17:32.245110 11515 leader_election.cc:288] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 170263bb913b4460a501597eb3b1cfdb (127.31.119.194:46593), eee526e1dc984352b0a297d99d6475a9 (127.31.119.193:39975)
I20260926 20:17:32.251176 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:32.245509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:39975 (local address 127.31.119.195:48361)
0926 20:17:32.245714 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 20:17:32.245732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:17:32.245842 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:17:32.247070 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:17:32.247073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:17:32.247095 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:17:32.247218 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:32.247226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:32.248305 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:32.248309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:32.249664 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:32.249677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:32.250726 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:32.250730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:32.250836 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:32.250862 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:17:32.250943 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:17:32.251024 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 20:17:32.252799 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:32.246134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:48361 (local address 127.31.119.193:39975)
0926 20:17:32.246258 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:32.246262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:32.246278 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:17:32.246334 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:32.246337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:32.246426 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:32.246508 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:17:32.247369 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:32.248160 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:32.249824 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:32.250424 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:32.250594 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:32.250657 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:32.252583 (+ 1926us) server_negotiation.cc:299] Negotiation successful
0926 20:17:32.252674 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:17:32.253497 11172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa2e865fc6475a9a195101faf0492e" candidate_uuid: "499ca67570af49e08f56da495b9e9675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee526e1dc984352b0a297d99d6475a9" is_pre_election: true
I20260926 20:17:32.253885 11172 raft_consensus.cc:2454] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 499ca67570af49e08f56da495b9e9675 in term 0.
I20260926 20:17:32.254621 11378 leader_election.cc:302] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499ca67570af49e08f56da495b9e9675, eee526e1dc984352b0a297d99d6475a9; no voters:
I20260926 20:17:32.254989 11515 raft_consensus.cc:2781] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:17:32.255077 11515 raft_consensus.cc:482] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:17:32.255123 11515 raft_consensus.cc:3037] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:17:32.256587 11515 raft_consensus.cc:504] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 } } peers { permanent_uuid: "499ca67570af49e08f56da495b9e9675" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40627 } } peers { permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 } }
I20260926 20:17:32.256855 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:32.245557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:51109 (local address 127.31.119.194:46593)
0926 20:17:32.246020 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:32.246025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:32.250754 (+ 4729us) server_negotiation.cc:407] Connection header received
0926 20:17:32.250965 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:32.250969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:32.251025 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:32.251075 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:17:32.251927 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:32.252703 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:32.255422 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:32.256079 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:32.256182 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:32.256234 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:32.256648 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 20:17:32.256731 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374}
I20260926 20:17:32.257004 11515 leader_election.cc:288] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [CANDIDATE]: Term 1 election: Requested vote from peers 170263bb913b4460a501597eb3b1cfdb (127.31.119.194:46593), eee526e1dc984352b0a297d99d6475a9 (127.31.119.193:39975)
I20260926 20:17:32.257223 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:32.246782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:46593 (local address 127.31.119.195:51109)
0926 20:17:32.250644 (+ 3862us) negotiation.cc:107] Waiting for socket to connect
0926 20:17:32.250661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:17:32.250837 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:17:32.251211 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:17:32.251214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:17:32.251230 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:17:32.251783 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:32.251791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:32.252851 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:32.252854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:32.255259 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:17:32.255270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:17:32.256379 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:17:32.256383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:17:32.256499 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:32.256523 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:17:32.256987 (+ 464us) client_negotiation.cc:240] Negotiation successful
0926 20:17:32.257068 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3772,"thread_start_us":119,"threads_started":1}
I20260926 20:17:32.258598 11301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa2e865fc6475a9a195101faf0492e" candidate_uuid: "499ca67570af49e08f56da495b9e9675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170263bb913b4460a501597eb3b1cfdb" is_pre_election: true
I20260926 20:17:32.258934 11301 raft_consensus.cc:2454] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 499ca67570af49e08f56da495b9e9675 in term 0.
I20260926 20:17:32.259292 11292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa2e865fc6475a9a195101faf0492e" candidate_uuid: "499ca67570af49e08f56da495b9e9675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170263bb913b4460a501597eb3b1cfdb"
I20260926 20:17:32.259397 11292 raft_consensus.cc:3037] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:17:32.260825 11292 raft_consensus.cc:2454] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 499ca67570af49e08f56da495b9e9675 in term 1.
I20260926 20:17:32.261731 11172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa2e865fc6475a9a195101faf0492e" candidate_uuid: "499ca67570af49e08f56da495b9e9675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee526e1dc984352b0a297d99d6475a9"
I20260926 20:17:32.261893 11172 raft_consensus.cc:3037] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:17:32.262516 11377 leader_election.cc:302] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170263bb913b4460a501597eb3b1cfdb, 499ca67570af49e08f56da495b9e9675; no voters:
I20260926 20:17:32.263291 11172 raft_consensus.cc:2454] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 499ca67570af49e08f56da495b9e9675 in term 1.
I20260926 20:17:32.263909 11515 raft_consensus.cc:2781] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:17:32.264165 11515 raft_consensus.cc:686] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 1 LEADER]: Becoming Leader. State: Replica: 499ca67570af49e08f56da495b9e9675, State: Running, Role: LEADER
I20260926 20:17:32.264575 11515 consensus_queue.cc:237] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 } } peers { permanent_uuid: "499ca67570af49e08f56da495b9e9675" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40627 } } peers { permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 } }
I20260926 20:17:32.268659 11005 catalog_manager.cc:5495] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 reported cstate change: term changed from 0 to 1, leader changed from <none> to 499ca67570af49e08f56da495b9e9675 (127.31.119.195). New cstate: current_term: 1 leader_uuid: "499ca67570af49e08f56da495b9e9675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "499ca67570af49e08f56da495b9e9675" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 40627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 } health_report { overall_health: UNKNOWN } } }
I20260926 20:17:32.351151 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:32.346984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.31.119.195:40627)
0926 20:17:32.347497 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 20:17:32.347501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:17:32.347518 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:17:32.347609 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:17:32.347614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:17:32.347684 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:17:32.347807 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:17:32.348066 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:32.348590 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:32.349575 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:17:32.349806 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:17:32.349920 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:17:32.350265 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:17:32.350734 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:17:32.350879 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:17:32.350920 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 20:17:32.350996 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412}
I20260926 20:17:32.357571 11292 raft_consensus.cc:1261] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb [term 1 FOLLOWER]: Refusing update from remote peer 499ca67570af49e08f56da495b9e9675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:17:32.357568 11172 raft_consensus.cc:1261] T 98aa2e865fc6475a9a195101faf0492e P eee526e1dc984352b0a297d99d6475a9 [term 1 FOLLOWER]: Refusing update from remote peer 499ca67570af49e08f56da495b9e9675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:17:32.358430 11515 consensus_queue.cc:1035] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee526e1dc984352b0a297d99d6475a9" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:17:32.358735 11520 consensus_queue.cc:1035] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "170263bb913b4460a501597eb3b1cfdb" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 46593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:17:32.367581 11530 mvcc.cc:205] Tried to move back new op lower bound from 7333698979248140288 to 7333698978880253952. Current Snapshot: MvccSnapshot[applied={T|T < 7333698979248140288}]
I20260926 20:17:32.369513 11533 mvcc.cc:205] Tried to move back new op lower bound from 7333698979248140288 to 7333698978880253952. Current Snapshot: MvccSnapshot[applied={T|T < 7333698979248140288}]
I20260926 20:17:32.381078 11001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:17:32.381280 11001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:17:32.383905 11001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0795dd2fb85a454f83a47f5460f5799a: Sending DeleteTablet for 3 replicas of tablet 98aa2e865fc6475a9a195101faf0492e
I20260926 20:17:32.384754 11283 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa2e865fc6475a9a195101faf0492e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:17:32 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:48942
I20260926 20:17:32.385202 11151 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa2e865fc6475a9a195101faf0492e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:17:32 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:43864
I20260926 20:17:32.385782 11424 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa2e865fc6475a9a195101faf0492e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:17:32 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:45142
I20260926 20:17:32.385921 11539 tablet_replica.cc:330] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: stopping tablet replica
I20260926 20:17:32.386281 11539 raft_consensus.cc:2229] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:17:32.386662 11539 raft_consensus.cc:2258] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:17:32.387123 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 11087
I20260926 20:17:32.388569 11539 ts_tablet_manager.cc:1905] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:17:32.391921 11539 ts_tablet_manager.cc:1918] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:17:32.392030 11539 log.cc:1196] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/98aa2e865fc6475a9a195101faf0492e
I20260926 20:17:32.391975 11541 tablet_replica.cc:330] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: stopping tablet replica
I20260926 20:17:32.392349 11539 ts_tablet_manager.cc:1939] T 98aa2e865fc6475a9a195101faf0492e P 170263bb913b4460a501597eb3b1cfdb: Deleting consensus metadata
I20260926 20:17:32.392592 11541 raft_consensus.cc:2229] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:17:32.393050 11541 raft_consensus.cc:2258] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:17:32.393718 10990 catalog_manager.cc:4841] TS 170263bb913b4460a501597eb3b1cfdb (127.31.119.194:46593): tablet 98aa2e865fc6475a9a195101faf0492e (table test-table [id=5f55d8cababc48f584c0bc188e4abd55]) successfully deleted
W20260926 20:17:32.398579 10991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:39975 (error 108)
I20260926 20:17:32.399101 11541 ts_tablet_manager.cc:1905] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:17:32.399415 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 11226
I20260926 20:17:32.402892 11541 ts_tablet_manager.cc:1918] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:17:32.403033 11541 log.cc:1196] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/98aa2e865fc6475a9a195101faf0492e
I20260926 20:17:32.403347 11541 ts_tablet_manager.cc:1939] T 98aa2e865fc6475a9a195101faf0492e P 499ca67570af49e08f56da495b9e9675: Deleting consensus metadata
I20260926 20:17:32.404572 10991 catalog_manager.cc:4841] TS 499ca67570af49e08f56da495b9e9675 (127.31.119.195:40627): tablet 98aa2e865fc6475a9a195101faf0492e (table test-table [id=5f55d8cababc48f584c0bc188e4abd55]) successfully deleted
I20260926 20:17:32.405329 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:17:32.404973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:39975 (local address 127.0.0.1:43876)
0926 20:17:32.405112 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:17:32.405189 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:39975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 20:17:32.405565 10991 catalog_manager.cc:4584] TS eee526e1dc984352b0a297d99d6475a9 (127.31.119.193:39975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98aa2e865fc6475a9a195101faf0492e: Network error: Client connection negotiation failed: client connection to 127.31.119.193:39975: connect: Connection refused (error 111)
I20260926 20:17:32.406679 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 11361
I20260926 20:17:32.413587 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 10974
I20260926 20:17:32.420609 32223 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:17:32.766551 32223 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:17:32.766722 32223 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:17:33.117640 32223 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:17:33.117 UTC [10560] LOG: received smart shutdown request
2026-09-26 20:17:33.121 UTC [10560] LOG: background worker "logical replication launcher" (PID 10569) exited with exit code 1
2026-09-26 20:17:33.122 UTC [10564] LOG: shutting down
2026-09-26 20:17:33.310 UTC [10560] LOG: database system is shut down
2026-09-26T20:17:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48054 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:17:33 dist-test-slave-vnbh krb5kdc[11551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:17:33 dist-test-slave-vnbh krb5kdc[11551](info): set up 2 sockets
Sep 26 20:17:33 dist-test-slave-vnbh krb5kdc[11551](info): commencing operation
krb5kdc: starting...
W20260926 20:17:34.494385 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:17:34 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:17:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:17:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.119.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.119.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/rangeradmin_127.31.119.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/rangeradmin_127.31.119.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.119.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.119.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/rangerlookup_127.31.119.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/rangerlookup_127.31.119.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.119.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.119.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/HTTP_127.31.119.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/HTTP_127.31.119.212@KRBTEST.COM.keytab.
I20260926 20:17:34.680114 32223 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYRmynJ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/postgres -l logfile start
2026-09-26 20:17:37.401 UTC [11581] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:17:37.401 UTC [11581] LOG: listening on IPv4 address "127.31.119.212", port 33325
2026-09-26 20:17:37.405 UTC [11581] LOG: listening on Unix socket "/tmp/.s.PGSQL.33325"
2026-09-26 20:17:37.426 UTC [11584] LOG: database system was shut down at 2026-09-26 20:17:35 UTC
2026-09-26 20:17:37.430 UTC [11581] LOG: database system is ready to accept connections
I20260926 20:17:38.398201 32223 mini_postgres.cc:96] Postgres bound to 33325
2026-09-26 20:17:38.404 UTC [11592] FATAL: database "slave" does not exist
127.31.119.212:33325 - accepting connections
I20260926 20:17:38.405505 32223 mini_ranger.cc:161] Starting Ranger...
I20260926 20:17:38.421263 32223 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:17:38.986591 32223 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:17:38.986688 32223 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-admin
2026-09-26 20:17:39,292 [I] DB FLAVOR :POSTGRES
2026-09-26 20:17:39,293 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:17:39,293 [I] Checking connection..
2026-09-26 20:17:39,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:17:39,548 [I] Checking connection passed.
2026-09-26 20:17:39,548 [I] --------- Verifying version history table ---------
2026-09-26 20:17:39,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:17:39,828 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:17:39,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:17:40,066 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:17:40,066 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:17:40,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:17:40.316 UTC [11656] WARNING: there is no transaction in progress
2026-09-26 20:17:40,323 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:17:40,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:17:40,581 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:17:40,582 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:17:40,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:17:40,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:17:41,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vnbh.c.gcp-upstream.internal','N') ;"
2026-09-26 20:17:41,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:17:41,640 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:17:41,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:17:41,913 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:17:41,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:17:42,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:17:42,449 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:17:42,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:17:43.491 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.509 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.526 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.543 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.561 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.581 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.647 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.656 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.674 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.685 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.695 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.711 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.721 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.728 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.736 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:43.744 UTC [11764] WARNING: there is no transaction in progress
2026-09-26 20:17:44,485 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:17:44,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:17:44,735 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:17:44,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:17:44,973 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:17:44,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:17:45,218 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:17:45,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:17:45,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:17:45,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vnbh.c.gcp-upstream.internal';"
2026-09-26 20:17:45,936 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:17:45,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:17:46,173 [I] DB_PATCHES have already been applied
I20260926 20:17:46.179369 32223 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYRmynJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:17:46.179438 32223 mini_ranger.cc:193] Using host: 127.31.119.212
I20260926 20:17:46.183112 32223 mini_ranger.cc:227] Ranger admin URL: http://127.31.119.212:35597
Sep 26, 2026 8:17:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:17:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:17:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:17:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:17:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:17:47.329385 32223 mini_ranger.cc:160] Time spent starting Ranger: real 8.924s user 0.005s sys 0.005s
Sep 26, 2026 8:18:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:18:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 8:18:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:18:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:18:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:18:09.102048 32223 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.119.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.119.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/rangerkms_127.31.119.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.119.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/rangerkms_127.31.119.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.119.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.119.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.119.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/HTTP_127.31.119.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.119.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/HTTP_127.31.119.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:18:09 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453889, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:18:09.192827 32223 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:18:09.202513 32223 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:18:10.199743 32223 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:18:10.199836 32223 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-kms
I20260926 20:18:10.199855 32223 mini_ranger_kms.cc:227] Using postgres at 127.31.119.212:33325
2026-09-26 20:18:10,374 [I] DB FLAVOR :POSTGRES
2026-09-26 20:18:10,374 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:18:10,374 [I] Checking connection
2026-09-26 20:18:10,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:18:10,613 [I] connection success
2026-09-26 20:18:10,614 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:18:10,614 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:18:10,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:18:10,859 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:18:10,860 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:18:10,860 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:18:10,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.119.212:33325/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:18:11,119 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:18:11.623169 32223 mini_ranger_kms.cc:310] Created kms service
I20260926 20:18:11.815840 32223 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:18:11.862610 32223 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:18:11 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453889, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:12.290393 32223 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:18:12.290550 32223 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYRmynJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYRmynJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:18:12.290576 32223 mini_ranger_kms.cc:240] Using host: 127.31.119.212
I20260926 20:18:12.294263 32223 mini_ranger_kms.cc:276] Ranger KMS PID: 11944
I20260926 20:18:12.294351 32223 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.119.212:56623
Sep 26, 2026 8:18:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:18:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:18:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:18:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:18:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:18:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 20:18:14.650244 32223 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.457s user 0.016s sys 0.141s
I20260926 20:18:14.650381 32223 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:18:14.650466 32223 mini_ranger_kms.cc:401] 127.31.119.212:56623/kms/v1/keys
Sep 26 20:18:14 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790453894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.119.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:18:15 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.119.212@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
Sep 26 20:18:15 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.119.212@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
Sep 26 20:18:17 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:18:18.033578 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:41225
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:41225
--ranger_config_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:56623/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:18.149430 11995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:18.149745 11995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:18.149787 11995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:18.154052 11995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:18:18.154129 11995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:18.154148 11995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:18.154165 11995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:18:18.154227 11995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:18:18.160379 11995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.119.212:56623/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:41225
--ranger_config_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:18.161621 11995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:18.162719 11995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:18.171900 12000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:18.172024 11995 server_base.cc:1034] running on GCE node
W20260926 20:18:18.171912 12003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:18.172322 12001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:18.172914 11995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:18.174198 11995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:18.175414 11995 hybrid_clock.cc:648] HybridClock initialized: now 1790453898175377 us; error 42 us; skew 500 ppm
Sep 26 20:18:18 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453898, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:18.182008 11995 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:18:18.183077 11995 webserver.cc:460] Webserver started at http://127.31.119.254:37429/ using document root <none> and password file <none>
I20260926 20:18:18.183669 11995 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:18.183740 11995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:18.183975 11995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:18:18 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453898, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:18.233780 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "5b498e92cd754cb0a5837f0e498bfe5f"
format_stamp: "Formatted at 2026-09-26 20:18:18 on dist-test-slave-vnbh"
server_key: "faf492d25b9b055e3896546e91bc36e5"
server_key_iv: "556794db60524b2990ff433b7d7fe4e6"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:18.234558 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "5b498e92cd754cb0a5837f0e498bfe5f"
format_stamp: "Formatted at 2026-09-26 20:18:18 on dist-test-slave-vnbh"
server_key: "faf492d25b9b055e3896546e91bc36e5"
server_key_iv: "556794db60524b2990ff433b7d7fe4e6"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:18.239653 11995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:18:18.257527 12010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:18.259235 11995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:18.259387 11995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "5b498e92cd754cb0a5837f0e498bfe5f"
format_stamp: "Formatted at 2026-09-26 20:18:18 on dist-test-slave-vnbh"
server_key: "faf492d25b9b055e3896546e91bc36e5"
server_key_iv: "556794db60524b2990ff433b7d7fe4e6"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:18.259536 11995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:18.283427 11995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:18.286541 11995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:18.286800 11995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:18.295271 11995 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:41225
I20260926 20:18:18.295272 12062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:41225 every 8 connection(s)
I20260926 20:18:18.296272 11995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:18:18.299253 12063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:18.302214 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 11995
I20260926 20:18:18.302384 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:18:18.302686 32223 external_mini_cluster.cc:1428] Setting key d0deb8f871b12f7412bc7e44bb961ccf
I20260926 20:18:18.305649 12063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: Bootstrap starting.
I20260926 20:18:18.307910 12063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:18.308658 12063 log.cc:824] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:18.310700 12063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: No bootstrap required, opened a new log
Sep 26 20:18:18 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:18.314487 12063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } }
I20260926 20:18:18.314714 12063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:18.314786 12063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b498e92cd754cb0a5837f0e498bfe5f, State: Initialized, Role: FOLLOWER
I20260926 20:18:18.315263 12063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } }
I20260926 20:18:18.315392 12063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:18:18.315477 12063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:18:18.315577 12063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:18.316478 12063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } }
I20260926 20:18:18.316834 12063 leader_election.cc:302] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b498e92cd754cb0a5837f0e498bfe5f; no voters:
I20260926 20:18:18.317145 12063 leader_election.cc:288] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:18:18.317276 12068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:18:18.317497 12068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 1 LEADER]: Becoming Leader. State: Replica: 5b498e92cd754cb0a5837f0e498bfe5f, State: Running, Role: LEADER
I20260926 20:18:18.317934 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:18.304194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.31.119.254:41225)
0926 20:18:18.304651 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:18.304666 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:18.304701 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:18:18.305510 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:18.305529 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:18.305926 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:18.306263 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:18.307107 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:18.308132 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:18.309089 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:18.309642 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:18.309822 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:18.312662 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:18.312668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:18.312682 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:18.312709 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:18.314469 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:18.315026 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:18.315032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:18.315045 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:18.315149 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:18.315447 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:18.315451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:18.315452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:18.315955 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:18.316158 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:18.316454 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 20:18:18.316872 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":161,"threads_started":1}
I20260926 20:18:18.317934 12068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } }
I20260926 20:18:18.318522 12063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:18:18.319133 12068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b498e92cd754cb0a5837f0e498bfe5f. Latest consensus state: current_term: 1 leader_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } } }
I20260926 20:18:18.319247 12068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:18.319764 12069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } } }
I20260926 20:18:18.319942 12069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:18.320813 12063 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:18:18.320920 12077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:18:18.327097 12077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:18:18.334275 12077 catalog_manager.cc:1347] Generated new cluster ID: 51d71b74dbd24043aaaff29077086503
I20260926 20:18:18.334460 12077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:18:18.353497 12077 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:18:18.355059 12077 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:18:18.365625 12077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: Generated new TSK 0
I20260926 20:18:18.366659 12077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:18:19.021199 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:56623/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41225
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:19.172024 12108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:19.172304 12108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:19.172350 12108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:19.176885 12108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:19.176980 12108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:19.177069 12108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:18:19.184511 12108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.119.212:56623/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:19.186254 12108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:19.187503 12108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:19.199105 12118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:19.199116 12116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:19.199234 12115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:19.199530 12108 server_base.cc:1034] running on GCE node
I20260926 20:18:19.200204 12108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:19.201084 12108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:19.202296 12108 hybrid_clock.cc:648] HybridClock initialized: now 1790453899202272 us; error 65 us; skew 500 ppm
Sep 26 20:18:19 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:19.205974 12108 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:18:19.206930 12108 webserver.cc:460] Webserver started at http://127.31.119.193:34179/ using document root <none> and password file <none>
I20260926 20:18:19.207525 12108 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:19.207587 12108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:19.207808 12108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:18:19 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:19.226454 12108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "ac7ef72f52134e8f96b6c31e5e35e842"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "4cc8807c28b51e50813827146933fcdf"
server_key_iv: "4021ebcf463aa8ed50de20729d5f365c"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:19.228154 12108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "ac7ef72f52134e8f96b6c31e5e35e842"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "4cc8807c28b51e50813827146933fcdf"
server_key_iv: "4021ebcf463aa8ed50de20729d5f365c"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:19.233764 12108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:18:19.253762 12125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:19.255724 12108 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 20:18:19.255882 12108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "ac7ef72f52134e8f96b6c31e5e35e842"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "4cc8807c28b51e50813827146933fcdf"
server_key_iv: "4021ebcf463aa8ed50de20729d5f365c"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:19.256019 12108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:19.292052 12108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:19.296098 12108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:19.296357 12108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:19.297080 12108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:19.298170 12108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:19.298292 12108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:19.298412 12108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:19.298485 12108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:19.309173 12108 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:40471
I20260926 20:18:19.310667 12108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:18:19.311439 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 12108
I20260926 20:18:19.311570 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:18:19.311875 32223 external_mini_cluster.cc:1428] Setting key 66e2aa56029f347aab120d3e4319d6f5
I20260926 20:18:19.312992 12240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:40471 every 8 connection(s)
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
Sep 26 20:18:19 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:19.334513 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:19.315301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41225 (local address 127.31.119.193:36375)
0926 20:18:19.317392 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:19.317449 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:19.318702 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:19.320424 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:19.320437 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:19.320920 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:19.321733 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:19.321768 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:19.323061 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:19.323066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:19.324510 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:19.324520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:19.325099 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:19.325105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:19.325362 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:19.326047 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:19.326064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:19.327927 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:19.330484 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:19.330494 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:19.330515 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:19.330892 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:19.331225 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:19.331231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:19.331248 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:19.331448 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:19.332013 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:19.332020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:19.332436 (+ 416us) client_negotiation.cc:769] Sending connection context
0926 20:18:19.332672 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 20:18:19.333081 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1801,"thread_start_us":149,"threads_started":1}
I20260926 20:18:19.335513 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:19.315997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:36375 (local address 127.31.119.254:41225)
0926 20:18:19.319804 (+ 3807us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:19.319815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:19.319834 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:18:19.319885 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:19.319891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:19.319957 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:19.320060 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:19.321952 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:19.322884 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:19.324661 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:19.324926 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:19.325055 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:19.328148 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:19.328154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:19.328160 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:19.328194 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:19.330338 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:19.331015 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:19.331022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:19.331026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:19.331121 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:19.331559 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:19.331565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:19.331570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:19.331842 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:19.331949 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:19.335070 (+ 3121us) server_negotiation.cc:299] Negotiation successful
0926 20:18:19.335307 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3648,"thread_start_us":99,"threads_started":1}
I20260926 20:18:19.337631 12241 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41225
I20260926 20:18:19.337913 12241 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:19.338526 12241 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
I20260926 20:18:19.340354 12021 ts_manager.cc:194] Registered new tserver with Master: ac7ef72f52134e8f96b6c31e5e35e842 (127.31.119.193:40471)
I20260926 20:18:19.341763 12021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:36375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:18:19.346482 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:56623/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41225
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:18:19 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:18:19.494793 12247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:19.495227 12247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:19.495311 12247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:19.501564 12247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:19.501668 12247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:19.501765 12247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:18:19.510227 12247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.119.212:56623/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:19.511912 12247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:19.513391 12247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:19.526481 12254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:19.526589 12256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:19.526660 12253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:19.530625 12247 server_base.cc:1034] running on GCE node
I20260926 20:18:19.531190 12247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:19.532028 12247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:19.533227 12247 hybrid_clock.cc:648] HybridClock initialized: now 1790453899533189 us; error 53 us; skew 500 ppm
Sep 26 20:18:19 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:19.537002 12247 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:18:19.538365 12247 webserver.cc:460] Webserver started at http://127.31.119.194:34075/ using document root <none> and password file <none>
I20260926 20:18:19.539165 12247 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:19.539284 12247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:19.539567 12247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:18:19 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:19.556480 12247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "581c54afbb5ee1c2843431ad4aae65ea"
server_key_iv: "647f4e86eefe8e3bbc2fc93df2f2f716"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:19.563038 12247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "581c54afbb5ee1c2843431ad4aae65ea"
server_key_iv: "647f4e86eefe8e3bbc2fc93df2f2f716"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:19.569070 12247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 20:18:19.589818 12263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:19.591863 12247 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260926 20:18:19.592125 12247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "581c54afbb5ee1c2843431ad4aae65ea"
server_key_iv: "647f4e86eefe8e3bbc2fc93df2f2f716"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:19.592393 12247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:19.622807 12247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:19.627544 12247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:19.627764 12247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:19.628579 12247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:19.629777 12247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:19.629850 12247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:19.629915 12247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:19.629951 12247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:19.642364 12247 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:37939
I20260926 20:18:19.642421 12376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:37939 every 8 connection(s)
I20260926 20:18:19.643842 12247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:18:19.648038 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 12247
I20260926 20:18:19.648159 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:18:19.648460 32223 external_mini_cluster.cc:1428] Setting key 72367e859174cbe8ae1e1b8760844fc0
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
Sep 26 20:18:19 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
I20260926 20:18:19.673110 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:19.646134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:39839 (local address 127.31.119.254:41225)
0926 20:18:19.646291 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:19.646296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:19.651886 (+ 5590us) server_negotiation.cc:407] Connection header received
0926 20:18:19.653212 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:19.653215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:19.653273 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:19.653359 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:19.655056 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:19.655941 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:19.658569 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:19.658993 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:19.659119 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:19.668376 (+ 9257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:19.668380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:19.668383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:19.668415 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:19.670417 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:19.671146 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:19.671149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:19.671151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:19.671239 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:19.671706 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:19.671710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:19.671711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:19.671909 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:19.671985 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:19.672753 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 20:18:19.672912 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:18:19.674202 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:19.649238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41225 (local address 127.31.119.194:39839)
0926 20:18:19.651706 (+ 2468us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:19.651775 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:19.652906 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:19.653804 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:19.653814 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:19.654207 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:19.654890 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:19.654908 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:19.657376 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:19.657380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:19.658431 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:19.658442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:19.663117 (+ 4675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:19.663123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:19.663358 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:19.664059 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:19.664077 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:19.668152 (+ 4075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:19.670577 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:19.670584 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:19.670605 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:19.671013 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:19.671349 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:19.671352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:19.671354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:19.671574 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:19.672043 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:19.672048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:19.672406 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 20:18:19.672633 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 20:18:19.673015 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2188,"thread_start_us":158,"threads_started":1}
I20260926 20:18:19.675625 12377 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41225
I20260926 20:18:19.675904 12377 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:19.676558 12377 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
I20260926 20:18:19.677882 12021 ts_manager.cc:194] Registered new tserver with Master: de0cbc0b8a66400f9e3d5054d6f2d2a6 (127.31.119.194:37939)
I20260926 20:18:19.678658 12021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:39839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:18:19.692185 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:56623/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41225
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:19.850988 12382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:19.851400 12382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:19.851464 12382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:19.858194 12382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:19.858291 12382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:19.858383 12382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:18:19.866859 12382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.119.212:56623/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
Sep 26 20:18:19 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:19.868306 12382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:19.869692 12382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:19.893525 12388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:19.893682 12387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:19.894115 12390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:19.895349 12382 server_base.cc:1034] running on GCE node
I20260926 20:18:19.895888 12382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:19.896811 12382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:19.897994 12382 hybrid_clock.cc:648] HybridClock initialized: now 1790453899897963 us; error 53 us; skew 500 ppm
Sep 26 20:18:19 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:19.901731 12382 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:18:19.903069 12382 webserver.cc:460] Webserver started at http://127.31.119.195:34811/ using document root <none> and password file <none>
I20260926 20:18:19.903918 12382 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:19.904078 12382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:19.904371 12382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:18:19 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:19.922318 12382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "118c5e39e93f4ba78fa5a421ec93ae25"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "bfe33cc936c3c03bb529888a34fa831c"
server_key_iv: "14ffc5ab0e298c88dfac0262009f7d3f"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:19.923244 12382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "118c5e39e93f4ba78fa5a421ec93ae25"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "bfe33cc936c3c03bb529888a34fa831c"
server_key_iv: "14ffc5ab0e298c88dfac0262009f7d3f"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:19.933473 12382 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260926 20:18:19.960304 12397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:19.962194 12382 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260926 20:18:19.962412 12382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "118c5e39e93f4ba78fa5a421ec93ae25"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "bfe33cc936c3c03bb529888a34fa831c"
server_key_iv: "14ffc5ab0e298c88dfac0262009f7d3f"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:19.962638 12382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:19.977406 12382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:19.980698 12382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:19.980892 12382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:19.981603 12382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:19.982597 12382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:19.982654 12382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:19.982703 12382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:19.982728 12382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:19.992977 12382 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:45057
I20260926 20:18:19.993023 12510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:45057 every 8 connection(s)
I20260926 20:18:19.994387 12382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:18:19.995875 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 12382
I20260926 20:18:19.996007 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:18:19.996340 32223 external_mini_cluster.cc:1428] Setting key 95c916e31ce9ea119f03a2a01ed0a936
Sep 26 20:18:20 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453899, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:20.016160 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:19.999333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:58919 (local address 127.31.119.254:41225)
0926 20:18:19.999462 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:19.999467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:20.001981 (+ 2514us) server_negotiation.cc:407] Connection header received
0926 20:18:20.003193 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:20.003197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:20.003253 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:20.003362 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:20.004928 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.005804 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.007549 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.007888 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.008002 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:20.011504 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:20.011507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:20.011510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:20.011540 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:20.013433 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:20.014189 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:20.014192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:20.014194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:20.014291 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:20.014763 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:20.014766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:20.014768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:20.014996 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:20.015091 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:20.015844 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 20:18:20.015992 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:18:20.017583 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:19.998055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41225 (local address 127.31.119.195:58919)
0926 20:18:20.001778 (+ 3723us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:20.001848 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:20.002933 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:20.003590 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:20.003601 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:20.003999 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:20.004745 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:20.004765 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.005961 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.005964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:20.007407 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:20.007418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.008191 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.008196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:20.008424 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:20.009489 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:20.009511 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:20.011302 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:20.013572 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:20.013580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:20.013605 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:20.014064 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:20.014400 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:20.014404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:20.014407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:20.014649 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:20.015117 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:20.015126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:20.015536 (+ 410us) client_negotiation.cc:769] Sending connection context
0926 20:18:20.015795 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 20:18:20.016226 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3426,"thread_start_us":184,"threads_started":1}
I20260926 20:18:20.018944 12511 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41225
I20260926 20:18:20.019255 12511 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:20.019892 12511 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
I20260926 20:18:20.026095 12021 ts_manager.cc:194] Registered new tserver with Master: 118c5e39e93f4ba78fa5a421ec93ae25 (127.31.119.195:45057)
I20260926 20:18:20.026764 12021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:58919
I20260926 20:18:20.034905 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:18:20.035585 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 12108
I20260926 20:18:20.041754 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 12247
I20260926 20:18:20.047822 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 12382
I20260926 20:18:20.054646 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 11995
I20260926 20:18:20.062093 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:41225
--webserver_interface=127.31.119.254
--webserver_port=37429
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:41225
--ranger_config_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:56623/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.119.212:56623/kms with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:20.177695 12514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:20.178018 12514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:20.178087 12514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:20.182399 12514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:18:20.182495 12514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:20.182531 12514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:20.182569 12514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:18:20.182603 12514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:18:20.188297 12514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.119.212:56623/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:41225
--ranger_config_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=37429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:20.189654 12514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:20.190740 12514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:20.199677 12522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:20.199683 12520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:20.199708 12519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:20.200057 12514 server_base.cc:1034] running on GCE node
I20260926 20:18:20.200682 12514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:20.201901 12514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:20.203065 12514 hybrid_clock.cc:648] HybridClock initialized: now 1790453900203049 us; error 30 us; skew 500 ppm
Sep 26 20:18:20 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:20.206053 12514 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:18:20.207093 12514 webserver.cc:460] Webserver started at http://127.31.119.254:37429/ using document root <none> and password file <none>
I20260926 20:18:20.207674 12514 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:20.207744 12514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:18:20 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:20.262167 12514 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:18:20.265090 12530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:20.266656 12514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:18:20.266899 12514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "5b498e92cd754cb0a5837f0e498bfe5f"
format_stamp: "Formatted at 2026-09-26 20:18:18 on dist-test-slave-vnbh"
server_key: "faf492d25b9b055e3896546e91bc36e5"
server_key_iv: "556794db60524b2990ff433b7d7fe4e6"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:20.267484 12514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:20.317392 12514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:20.320514 12514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:20.320734 12514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:20.328598 12514 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:41225
I20260926 20:18:20.328603 12582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:41225 every 8 connection(s)
I20260926 20:18:20.329977 12514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:18:20.331043 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 12514
I20260926 20:18:20.331600 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:56623/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:40471
--local_ip_for_outbound_sockets=127.31.119.193
--tserver_master_addrs=127.31.119.254:41225
--webserver_port=34179
--webserver_interface=127.31.119.193
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.119.212:56623/kms with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:18:20.336035 12583 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:18:20.338049 12583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: Bootstrap starting.
I20260926 20:18:20.356526 12583 log.cc:824] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:20.363818 12583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:18:20.364287 12583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: Bootstrap complete.
I20260926 20:18:20.368930 12583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } }
I20260926 20:18:20.369423 12583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b498e92cd754cb0a5837f0e498bfe5f, State: Initialized, Role: FOLLOWER
I20260926 20:18:20.369964 12583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } }
I20260926 20:18:20.370083 12583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:18:20.370179 12583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:18:20.370288 12583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:18:20.372447 12583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } }
I20260926 20:18:20.372802 12583 leader_election.cc:302] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b498e92cd754cb0a5837f0e498bfe5f; no voters:
I20260926 20:18:20.373169 12583 leader_election.cc:288] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:18:20.373322 12587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:18:20.373592 12587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [term 2 LEADER]: Becoming Leader. State: Replica: 5b498e92cd754cb0a5837f0e498bfe5f, State: Running, Role: LEADER
I20260926 20:18:20.373970 12587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } }
I20260926 20:18:20.374524 12583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:18:20.375554 12588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } } }
I20260926 20:18:20.375708 12588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:20.376111 12596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:18:20.376060 12589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b498e92cd754cb0a5837f0e498bfe5f. Latest consensus state: current_term: 2 leader_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b498e92cd754cb0a5837f0e498bfe5f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41225 } } }
I20260926 20:18:20.376251 12589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:20.379721 12596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:18:20.380367 12596 catalog_manager.cc:1259] Loaded cluster ID: 51d71b74dbd24043aaaff29077086503
I20260926 20:18:20.380375 12583 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:18:20.380493 12596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:18:20.382786 12596 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:18:20.383422 12596 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: Loaded TSK: 0
I20260926 20:18:20.384609 12596 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:18:20.474151 12585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:20.474630 12585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:20.474766 12585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:20.480877 12585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:20.480966 12585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:20.481057 12585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:18:20.489015 12585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.119.212:56623/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:40471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=34179
--tserver_master_addrs=127.31.119.254:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:20.490465 12585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:20.491928 12585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:18:20.507387 12585 server_base.cc:1034] running on GCE node
W20260926 20:18:20.508628 12623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:20.509425 12621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:20.509676 12620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:20.510242 12585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:20.511091 12585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:20.512292 12585 hybrid_clock.cc:648] HybridClock initialized: now 1790453900512259 us; error 57 us; skew 500 ppm
Sep 26 20:18:20 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:20.517128 12585 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:18:20.518530 12585 webserver.cc:460] Webserver started at http://127.31.119.193:34179/ using document root <none> and password file <none>
I20260926 20:18:20.519411 12585 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:20.519532 12585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:18:20 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:20.555613 12585 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:18:20.558833 12631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:20.561005 12585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:18:20.561364 12585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "ac7ef72f52134e8f96b6c31e5e35e842"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "4cc8807c28b51e50813827146933fcdf"
server_key_iv: "4021ebcf463aa8ed50de20729d5f365c"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:20.562999 12585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:20.594874 12585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:20.599232 12585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:20.599480 12585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:20.600229 12585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:20.601451 12585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:20.601575 12585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:20.601660 12585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:20.601691 12585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:20.615979 12585 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:40471
I20260926 20:18:20.617658 12585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:18:20.618402 12744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:40471 every 8 connection(s)
I20260926 20:18:20.620752 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 12585
I20260926 20:18:20.621356 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:56623/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:37939
--local_ip_for_outbound_sockets=127.31.119.194
--tserver_master_addrs=127.31.119.254:41225
--webserver_port=34075
--webserver_interface=127.31.119.194
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.119.212:56623/kms with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:18:20 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:20.644351 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:20.620740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41225 (local address 127.31.119.193:47271)
0926 20:18:20.622530 (+ 1790us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:20.622599 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:20.624005 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:20.626356 (+ 2351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:20.626370 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:20.626795 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:20.627635 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:20.627663 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.630223 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.630230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:20.632137 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:20.632158 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.633997 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.634003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:20.634285 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:20.635114 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:20.635145 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:20.637101 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:20.640001 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:20.640008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:20.640026 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:20.640424 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:20.640881 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:20.640885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:20.640887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:20.641068 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:20.642035 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:20.642043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:20.642381 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 20:18:20.642608 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 20:18:20.643068 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"thread_start_us":169,"threads_started":1}
I20260926 20:18:20.646450 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:20.620649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:47271 (local address 127.31.119.254:41225)
0926 20:18:20.621175 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:20.621195 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:20.622768 (+ 1573us) server_negotiation.cc:407] Connection header received
0926 20:18:20.624757 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:20.624776 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:20.625274 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:20.625700 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:20.627923 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.629981 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.632364 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.633428 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.633660 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:20.637436 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:20.637445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:20.637468 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:20.637507 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:20.639814 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:20.640579 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:20.640587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:20.640593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:20.640753 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:20.641180 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:20.641184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:20.641186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:20.641769 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:20.641965 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:20.644999 (+ 3034us) server_negotiation.cc:299] Negotiation successful
0926 20:18:20.645382 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":155,"threads_started":1}
I20260926 20:18:20.650384 12745 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41225
I20260926 20:18:20.650668 12745 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:20.651450 12745 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
W20260926 20:18:20.653961 12745 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:41225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:18:20.654102 12745 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:20.654320 12745 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
I20260926 20:18:20.655223 12745 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:20.655438 12745 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
W20260926 20:18:20.656235 12745 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:18:20.774308 12749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:20.774665 12749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:20.774731 12749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:20.779361 12749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:20.779421 12749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:20.779482 12749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:18:20.785595 12749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.119.212:56623/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:37939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=34075
--tserver_master_addrs=127.31.119.254:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:20.786885 12749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:20.788053 12749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:20.798494 12754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:20.799017 12757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:20.798496 12755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:20.800393 12749 server_base.cc:1034] running on GCE node
I20260926 20:18:20.801028 12749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:20.802101 12749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:20.803385 12749 hybrid_clock.cc:648] HybridClock initialized: now 1790453900803356 us; error 51 us; skew 500 ppm
Sep 26 20:18:20 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:20.806874 12749 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:18:20.808162 12749 webserver.cc:460] Webserver started at http://127.31.119.194:34075/ using document root <none> and password file <none>
I20260926 20:18:20.808802 12749 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:20.808924 12749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:18:20 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:20.829613 12749 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:18:20.834049 12765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:20.835584 12749 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 20:18:20.835742 12749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "581c54afbb5ee1c2843431ad4aae65ea"
server_key_iv: "647f4e86eefe8e3bbc2fc93df2f2f716"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:20.836360 12749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:20.853693 12749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:20.858076 12749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:20.858311 12749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:20.859071 12749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:20.860178 12749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:20.860256 12749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:20.860322 12749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:20.860368 12749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:20.872022 12749 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:37939
I20260926 20:18:20.872061 12878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:37939 every 8 connection(s)
I20260926 20:18:20.873479 12749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:18:20.880664 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 12749
I20260926 20:18:20.881192 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.119.212:56623/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:45057
--local_ip_for_outbound_sockets=127.31.119.195
--tserver_master_addrs=127.31.119.254:41225
--webserver_port=34811
--webserver_interface=127.31.119.195
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.119.212:56623/kms with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:18:20 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453900, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:20.895282 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:20.875764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:33299 (local address 127.31.119.254:41225)
0926 20:18:20.875902 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:20.875907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:20.876946 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 20:18:20.878504 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:20.878508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:20.878571 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:20.878630 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:20.880596 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.884210 (+ 3614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.886170 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.886572 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.886659 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:20.890234 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:20.890238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:20.890241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:20.890278 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:20.892263 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:20.893065 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:20.893073 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:20.893076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:20.893177 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:20.893695 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:20.893700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:20.893703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:20.893982 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:20.894080 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:20.894955 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 20:18:20.895103 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:18:20.896915 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:20.876287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41225 (local address 127.31.119.194:33299)
0926 20:18:20.876788 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:20.876841 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:20.878224 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:20.879327 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:20.879337 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:20.879698 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:20.880402 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:20.880423 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.884424 (+ 4001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.884427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:20.886005 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:20.886019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:20.887106 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:20.887110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:20.887338 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:20.888094 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:20.888116 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:20.890012 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:20.892458 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:20.892468 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:20.892490 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:20.892923 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:20.893346 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:20.893350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:20.893354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:20.893568 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:20.894156 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:20.894166 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:20.894577 (+ 411us) client_negotiation.cc:769] Sending connection context
0926 20:18:20.894828 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 20:18:20.895259 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":169,"threads_started":1}
I20260926 20:18:20.899717 12879 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41225
I20260926 20:18:20.899988 12879 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:20.900601 12879 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
W20260926 20:18:20.902539 12879 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:41225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:18:20.902668 12879 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:20.902853 12879 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
I20260926 20:18:20.905117 12879 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:20.905350 12879 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
W20260926 20:18:20.906111 12879 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:18:21.030061 12882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:21.030354 12882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:21.030397 12882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:21.035143 12882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:21.035238 12882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:21.035346 12882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:18:21.042461 12882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.119.212:56623/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:45057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=34811
--tserver_master_addrs=127.31.119.254:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:21.043920 12882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:21.045122 12882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:21.058073 12888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:21.058164 12889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:21.058073 12891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:21.058626 12882 server_base.cc:1034] running on GCE node
I20260926 20:18:21.061529 12882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:21.062420 12882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:21.063627 12882 hybrid_clock.cc:648] HybridClock initialized: now 1790453901063589 us; error 53 us; skew 500 ppm
Sep 26 20:18:21 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453901, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:21.067394 12882 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:18:21.068542 12882 webserver.cc:460] Webserver started at http://127.31.119.195:34811/ using document root <none> and password file <none>
I20260926 20:18:21.069319 12882 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:21.069432 12882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:18:21 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453901, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:21.088331 12882 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 20:18:21.091437 12899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:21.092967 12882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:18:21.093114 12882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "118c5e39e93f4ba78fa5a421ec93ae25"
format_stamp: "Formatted at 2026-09-26 20:18:19 on dist-test-slave-vnbh"
server_key: "bfe33cc936c3c03bb529888a34fa831c"
server_key_iv: "14ffc5ab0e298c88dfac0262009f7d3f"
server_key_version: "kuduclusterkey@0"
I20260926 20:18:21.093765 12882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:21.108575 12882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:21.112198 12882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:21.112421 12882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:21.113097 12882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:21.114078 12882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:21.114151 12882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:21.114235 12882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:21.114269 12882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:21.126026 12882 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:45057
I20260926 20:18:21.127143 12882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:18:21.128774 13012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:45057 every 8 connection(s)
I20260926 20:18:21.135330 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 12882
Sep 26 20:18:21 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453901, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:21.144366 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:21.130961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:39517 (local address 127.31.119.254:41225)
0926 20:18:21.131136 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:21.131140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:21.131158 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:18:21.131638 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:21.131642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:21.131701 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:21.131780 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:21.132974 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:21.133612 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:21.135054 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:21.135394 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:21.135497 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:21.139251 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:21.139256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:21.139259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:21.139292 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:21.141572 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:21.142318 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:21.142323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:21.142334 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:21.142451 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:21.142960 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:21.142964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:21.142966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:21.143183 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:21.143282 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:21.143975 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 20:18:21.144164 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 20:18:21.145541 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:21.129829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:41225 (local address 127.31.119.195:39517)
0926 20:18:21.130326 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:21.130380 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:21.131407 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:21.131971 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:21.131979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:21.132250 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:21.132747 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:21.132760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:21.133746 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:21.133750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:21.134865 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:21.134877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:21.135720 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:21.135727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:21.135970 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:21.136924 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:21.136950 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:21.138990 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:21.141804 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:21.141812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:21.141834 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:21.142188 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:21.142583 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:21.142587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:21.142589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:21.142807 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:21.143377 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:21.143384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:21.143684 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 20:18:21.143917 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 20:18:21.144316 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":145,"threads_started":1}
I20260926 20:18:21.147039 13013 heartbeater.cc:347] Connected to a master server at 127.31.119.254:41225
I20260926 20:18:21.147330 13013 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:21.148051 13013 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
W20260926 20:18:21.150048 13013 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:41225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:18:21.150208 13013 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:21.150444 13013 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
I20260926 20:18:21.151419 13013 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:21.151630 13013 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
W20260926 20:18:21.152472 13013 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:18:21.153687 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:21.136613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.31.119.254:41225)
0926 20:18:21.137928 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:21.137940 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:21.139802 (+ 1862us) server_negotiation.cc:407] Connection header received
0926 20:18:21.140408 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:21.140412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:21.140471 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:21.142314 (+ 1843us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:21.142406 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:21.143252 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:21.146794 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:21.147193 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:21.147300 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:21.149393 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:21.149399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:21.149405 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:21.149432 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:21.151470 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:21.152024 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:21.152027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:21.152029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:21.152125 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:21.152503 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:21.152506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:21.152508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:21.152782 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:21.153009 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:21.153233 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 20:18:21.153461 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":109,"threads_started":1}
Sep 26 20:18:21 dist-test-slave-vnbh krb5kdc[11551](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790453901, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:21.656491 12745 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:21.656834 12745 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
I20260926 20:18:21.658138 12547 ts_manager.cc:194] Registered new tserver with Master: ac7ef72f52134e8f96b6c31e5e35e842 (127.31.119.193:40471)
I20260926 20:18:21.659493 12547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:47271
I20260926 20:18:21.906375 12879 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:21.906697 12879 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
I20260926 20:18:21.907424 12547 ts_manager.cc:194] Registered new tserver with Master: de0cbc0b8a66400f9e3d5054d6f2d2a6 (127.31.119.194:37939)
I20260926 20:18:21.908277 12547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:33299
Sep 26 20:18:22 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453901, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:22.152730 13013 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:22.153118 13013 heartbeater.cc:511] Master 127.31.119.254:41225 requested a full tablet report, sending...
I20260926 20:18:22.154227 12547 ts_manager.cc:194] Registered new tserver with Master: 118c5e39e93f4ba78fa5a421ec93ae25 (127.31.119.195:45057)
I20260926 20:18:22.155014 12547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:39517
I20260926 20:18:22.157927 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:18:22.169921 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.159497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.31.119.254:41225)
0926 20:18:22.159723 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:22.159728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:22.160042 (+ 314us) server_negotiation.cc:407] Connection header received
0926 20:18:22.160178 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:22.160182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:22.160231 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:22.160321 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:22.161241 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.162059 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.163253 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.163605 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.163715 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.165656 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:22.165661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:22.165664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:22.165695 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:22.167719 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:22.168263 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:22.168267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:22.168269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:22.168372 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:22.168670 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:22.168673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:22.168675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:22.168925 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:22.169071 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:22.169368 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 20:18:22.169711 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260926 20:18:22.174532 12547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:18:22.179468 12547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:18:22.200129 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.192246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.31.119.193:40471)
0926 20:18:22.193175 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:22.193182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:22.194515 (+ 1333us) server_negotiation.cc:407] Connection header received
0926 20:18:22.194624 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:22.194630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:22.194850 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:22.194989 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:22.195447 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.196540 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.198188 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.199076 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.199187 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.199316 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:22.199891 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 20:18:22.199990 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":105,"threads_started":1}
I20260926 20:18:22.200603 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.191901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40471 (local address 127.0.0.1:47770)
0926 20:18:22.194398 (+ 2497us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:22.194411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:22.194499 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:22.195170 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:22.195176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:22.195198 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:22.195318 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:22.195324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.196668 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.196672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:22.198049 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:22.198060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.199375 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.199379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:22.199486 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.199522 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:18:22.199627 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:18:22.199742 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2435,"mutex_wait_us":599,"thread_start_us":78,"threads_started":1}
I20260926 20:18:22.202499 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.193521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:37939 (local address 127.0.0.1:49142)
0926 20:18:22.195637 (+ 2116us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:22.195649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:22.195735 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:22.196498 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:22.196501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:22.196519 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:22.196621 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:22.196631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.198790 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.198794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:22.200036 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:22.200047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.202134 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.202138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:22.202223 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.202248 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:18:22.202318 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:18:22.202383 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2057,"thread_start_us":95,"threads_started":1}
I20260926 20:18:22.203223 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.193636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.31.119.194:37939)
0926 20:18:22.194719 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:22.194726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:22.195850 (+ 1124us) server_negotiation.cc:407] Connection header received
0926 20:18:22.195999 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:22.196003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:22.196202 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:22.196350 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:22.196761 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.198650 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.200179 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.202000 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.202130 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.202251 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:22.202988 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 20:18:22.203090 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":127,"threads_started":1}
I20260926 20:18:22.204385 12678 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9b64766a5149779623fb8af746210b (DEFAULT_TABLE table=test-table [id=dce8e7b18c814cb1993ae9f716b92877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:22.205758 12678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9b64766a5149779623fb8af746210b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:22.206472 12811 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9b64766a5149779623fb8af746210b (DEFAULT_TABLE table=test-table [id=dce8e7b18c814cb1993ae9f716b92877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:22.207544 12811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9b64766a5149779623fb8af746210b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:22.208859 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.191899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:45057 (local address 127.0.0.1:60458)
0926 20:18:22.193975 (+ 2076us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:22.194027 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:22.194156 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:22.202987 (+ 8831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:22.202990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:22.203007 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:22.203098 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:22.203104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.206241 (+ 3137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.206245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:22.207344 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:22.207358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.208443 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.208447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:22.208541 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.208570 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 20:18:22.208623 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 20:18:22.208700 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"thread_start_us":108,"threads_started":1}
I20260926 20:18:22.210549 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.192602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.31.119.195:45057)
0926 20:18:22.202506 (+ 9904us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:22.202513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:22.202539 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:18:22.202615 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:22.202620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:22.202831 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:22.203243 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:22.203381 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.206106 (+ 2725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.207516 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.208298 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.209157 (+ 859us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.209316 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:22.209378 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 20:18:22.210402 (+ 1024us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9740,"thread_start_us":87,"threads_started":1}
I20260926 20:18:22.213655 12947 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9b64766a5149779623fb8af746210b (DEFAULT_TABLE table=test-table [id=dce8e7b18c814cb1993ae9f716b92877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:22.214823 12947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9b64766a5149779623fb8af746210b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:22.224850 13046 tablet_bootstrap.cc:492] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: Bootstrap starting.
I20260926 20:18:22.227614 13046 tablet_bootstrap.cc:654] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:22.228572 13046 log.cc:824] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:22.229717 13047 tablet_bootstrap.cc:492] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842: Bootstrap starting.
I20260926 20:18:22.232307 13047 tablet_bootstrap.cc:654] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:22.232838 13048 tablet_bootstrap.cc:492] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: Bootstrap starting.
I20260926 20:18:22.235466 13048 tablet_bootstrap.cc:654] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:22.235903 13046 tablet_bootstrap.cc:492] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: No bootstrap required, opened a new log
I20260926 20:18:22.236131 13046 ts_tablet_manager.cc:1397] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260926 20:18:22.236626 13047 log.cc:824] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:22.237638 13048 log.cc:824] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:22.240463 13046 raft_consensus.cc:348] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 } } peers { permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 } } peers { permanent_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37939 } }
I20260926 20:18:22.240903 13048 tablet_bootstrap.cc:492] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: No bootstrap required, opened a new log
I20260926 20:18:22.240916 13046 raft_consensus.cc:374] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:22.241012 13046 raft_consensus.cc:729] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de0cbc0b8a66400f9e3d5054d6f2d2a6, State: Initialized, Role: FOLLOWER
I20260926 20:18:22.241132 13048 ts_tablet_manager.cc:1397] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 20:18:22.241675 13046 consensus_queue.cc:260] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 } } peers { permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 } } peers { permanent_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37939 } }
I20260926 20:18:22.243908 13046 ts_tablet_manager.cc:1428] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260926 20:18:22.244284 12879 heartbeater.cc:503] Master 127.31.119.254:41225 was elected leader, sending a full tablet report...
I20260926 20:18:22.245590 13048 raft_consensus.cc:348] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 } } peers { permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 } } peers { permanent_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37939 } }
I20260926 20:18:22.247910 13047 tablet_bootstrap.cc:492] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842: No bootstrap required, opened a new log
I20260926 20:18:22.248152 13047 ts_tablet_manager.cc:1397] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260926 20:18:22.252348 13048 raft_consensus.cc:374] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:22.252519 13048 raft_consensus.cc:729] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118c5e39e93f4ba78fa5a421ec93ae25, State: Initialized, Role: FOLLOWER
I20260926 20:18:22.252416 13047 raft_consensus.cc:348] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 } } peers { permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 } } peers { permanent_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37939 } }
I20260926 20:18:22.252833 13047 raft_consensus.cc:374] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:22.252898 13047 raft_consensus.cc:729] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac7ef72f52134e8f96b6c31e5e35e842, State: Initialized, Role: FOLLOWER
I20260926 20:18:22.253177 13048 consensus_queue.cc:260] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 } } peers { permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 } } peers { permanent_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37939 } }
I20260926 20:18:22.253407 13047 consensus_queue.cc:260] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 } } peers { permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 } } peers { permanent_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37939 } }
I20260926 20:18:22.254669 13047 ts_tablet_manager.cc:1428] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:18:22.254669 13048 ts_tablet_manager.cc:1428] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260926 20:18:22.255052 12745 heartbeater.cc:503] Master 127.31.119.254:41225 was elected leader, sending a full tablet report...
I20260926 20:18:22.255424 13013 heartbeater.cc:503] Master 127.31.119.254:41225 was elected leader, sending a full tablet report...
I20260926 20:18:22.261186 13052 raft_consensus.cc:482] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:18:22.261427 13052 raft_consensus.cc:504] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 } } peers { permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 } } peers { permanent_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37939 } }
I20260926 20:18:22.262632 13052 leader_election.cc:288] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac7ef72f52134e8f96b6c31e5e35e842 (127.31.119.193:40471), 118c5e39e93f4ba78fa5a421ec93ae25 (127.31.119.195:45057)
I20260926 20:18:22.270483 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.262898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:43701 (local address 127.31.119.193:40471)
0926 20:18:22.263028 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:22.263033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:22.263600 (+ 567us) server_negotiation.cc:407] Connection header received
0926 20:18:22.263831 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:22.263835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:22.263885 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:22.263937 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:22.264405 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.265201 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.266865 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.267456 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.267542 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.267598 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:22.270256 (+ 2658us) server_negotiation.cc:299] Negotiation successful
0926 20:18:22.270346 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:18:22.270992 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.262833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40471 (local address 127.31.119.194:43701)
0926 20:18:22.263507 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:22.263523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:22.263683 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:22.264090 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:22.264092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:22.264113 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:22.264263 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:22.264271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.265382 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.265385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:22.266713 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:22.266727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.267695 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.267699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:22.270110 (+ 2411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.270136 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:18:22.270648 (+ 512us) client_negotiation.cc:240] Negotiation successful
0926 20:18:22.270733 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":2241,"thread_start_us":128,"threads_started":1}
I20260926 20:18:22.272128 12686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9b64766a5149779623fb8af746210b" candidate_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" is_pre_election: true
I20260926 20:18:22.272514 12686 raft_consensus.cc:2454] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de0cbc0b8a66400f9e3d5054d6f2d2a6 in term 0.
I20260926 20:18:22.273084 12768 leader_election.cc:302] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac7ef72f52134e8f96b6c31e5e35e842, de0cbc0b8a66400f9e3d5054d6f2d2a6; no voters:
I20260926 20:18:22.273480 13052 raft_consensus.cc:2781] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:18:22.273583 13052 raft_consensus.cc:482] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:18:22.273650 13052 raft_consensus.cc:3037] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:22.275189 13052 raft_consensus.cc:504] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 } } peers { permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 } } peers { permanent_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37939 } }
I20260926 20:18:22.275650 13052 leader_election.cc:288] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [CANDIDATE]: Term 1 election: Requested vote from peers ac7ef72f52134e8f96b6c31e5e35e842 (127.31.119.193:40471), 118c5e39e93f4ba78fa5a421ec93ae25 (127.31.119.195:45057)
I20260926 20:18:22.276333 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.263614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:38887 (local address 127.31.119.195:45057)
0926 20:18:22.263735 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:22.263739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:22.266731 (+ 2992us) server_negotiation.cc:407] Connection header received
0926 20:18:22.267194 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:22.267198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:22.267252 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:22.267312 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:22.272281 (+ 4969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.273075 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.274979 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.275603 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.275702 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.275753 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:22.276128 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 20:18:22.276209 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:18:22.276324 12686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9b64766a5149779623fb8af746210b" candidate_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac7ef72f52134e8f96b6c31e5e35e842"
I20260926 20:18:22.276466 12686 raft_consensus.cc:3037] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:22.276693 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.263232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:45057 (local address 127.31.119.194:38887)
0926 20:18:22.266625 (+ 3393us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:22.266641 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:22.266806 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:22.267451 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:22.267453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:22.267473 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:22.269820 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:22.269839 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.273535 (+ 3696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.273538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:22.274772 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:22.274784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.275884 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.275888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:22.275990 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.276013 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:18:22.276479 (+ 466us) client_negotiation.cc:240] Negotiation successful
0926 20:18:22.276559 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3315,"thread_start_us":109,"threads_started":1}
I20260926 20:18:22.277787 12959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9b64766a5149779623fb8af746210b" candidate_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" is_pre_election: true
I20260926 20:18:22.278120 12959 raft_consensus.cc:2454] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de0cbc0b8a66400f9e3d5054d6f2d2a6 in term 0.
I20260926 20:18:22.278138 12686 raft_consensus.cc:2454] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de0cbc0b8a66400f9e3d5054d6f2d2a6 in term 1.
I20260926 20:18:22.278435 12958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9b64766a5149779623fb8af746210b" candidate_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "118c5e39e93f4ba78fa5a421ec93ae25"
I20260926 20:18:22.278506 12768 leader_election.cc:302] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac7ef72f52134e8f96b6c31e5e35e842, de0cbc0b8a66400f9e3d5054d6f2d2a6; no voters:
I20260926 20:18:22.278540 12958 raft_consensus.cc:3037] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:22.278713 13052 raft_consensus.cc:2781] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:18:22.278968 13052 raft_consensus.cc:686] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 1 LEADER]: Becoming Leader. State: Replica: de0cbc0b8a66400f9e3d5054d6f2d2a6, State: Running, Role: LEADER
I20260926 20:18:22.279371 13052 consensus_queue.cc:237] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 } } peers { permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 } } peers { permanent_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37939 } }
I20260926 20:18:22.279779 12958 raft_consensus.cc:2454] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de0cbc0b8a66400f9e3d5054d6f2d2a6 in term 1.
I20260926 20:18:22.283135 12546 catalog_manager.cc:5495] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to de0cbc0b8a66400f9e3d5054d6f2d2a6 (127.31.119.194). New cstate: current_term: 1 leader_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de0cbc0b8a66400f9e3d5054d6f2d2a6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37939 } health_report { overall_health: HEALTHY } } }
Sep 26 20:18:22 dist-test-slave-vnbh krb5kdc[11551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453901, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for HTTP/127.31.119.212@KRBTEST.COM
I20260926 20:18:22.309886 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.304121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.31.119.194:37939)
0926 20:18:22.304427 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:22.304432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:22.304448 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:18:22.304594 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:22.304598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:22.304694 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:22.304870 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:18:22.305348 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.306184 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.307487 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:22.307791 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:22.307920 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:22.308353 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:18:22.308979 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:18:22.309107 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:22.309634 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 20:18:22.309708 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 20:18:22.317343 12958 raft_consensus.cc:1261] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25 [term 1 FOLLOWER]: Refusing update from remote peer de0cbc0b8a66400f9e3d5054d6f2d2a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:18:22.317365 12686 raft_consensus.cc:1261] T 7b9b64766a5149779623fb8af746210b P ac7ef72f52134e8f96b6c31e5e35e842 [term 1 FOLLOWER]: Refusing update from remote peer de0cbc0b8a66400f9e3d5054d6f2d2a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:18:22.318162 13052 consensus_queue.cc:1035] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac7ef72f52134e8f96b6c31e5e35e842" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:18:22.318737 13057 consensus_queue.cc:1035] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "118c5e39e93f4ba78fa5a421ec93ae25" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 45057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:18:22.345605 13063 mvcc.cc:205] Tried to move back new op lower bound from 7333699183882502144 to 7333699183739842560. Current Snapshot: MvccSnapshot[applied={T|T < 7333699183882502144}]
I20260926 20:18:22.345858 13062 mvcc.cc:205] Tried to move back new op lower bound from 7333699183882502144 to 7333699183739842560. Current Snapshot: MvccSnapshot[applied={T|T < 7333699183882502144}]
I20260926 20:18:22.346069 13061 mvcc.cc:205] Tried to move back new op lower bound from 7333699183882502144 to 7333699183739842560. Current Snapshot: MvccSnapshot[applied={T|T < 7333699183882502144}]
I20260926 20:18:22.368547 12547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:18:22.368763 12547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:18:22.373167 12547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b498e92cd754cb0a5837f0e498bfe5f: Sending DeleteTablet for 3 replicas of tablet 7b9b64766a5149779623fb8af746210b
I20260926 20:18:22.374193 12947 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9b64766a5149779623fb8af746210b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:22 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:60458
I20260926 20:18:22.374176 12678 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9b64766a5149779623fb8af746210b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:22 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:47770
I20260926 20:18:22.375376 12811 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9b64766a5149779623fb8af746210b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:22 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:49142
I20260926 20:18:22.376968 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 12585
I20260926 20:18:22.383394 13073 tablet_replica.cc:330] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: stopping tablet replica
I20260926 20:18:22.383910 13073 raft_consensus.cc:2229] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:18:22.384482 13073 raft_consensus.cc:2258] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:18:22.384807 13071 tablet_replica.cc:330] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: stopping tablet replica
I20260926 20:18:22.385159 13071 raft_consensus.cc:2229] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:18:22.386912 13073 ts_tablet_manager.cc:1905] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:18:22.387343 13071 raft_consensus.cc:2258] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:18:22.388881 13071 ts_tablet_manager.cc:1905] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:18:22.392056 13073 ts_tablet_manager.cc:1918] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:18:22.394872 13071 ts_tablet_manager.cc:1918] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:18:22.395051 13073 log.cc:1196] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/7b9b64766a5149779623fb8af746210b
I20260926 20:18:22.395545 13073 ts_tablet_manager.cc:1939] T 7b9b64766a5149779623fb8af746210b P de0cbc0b8a66400f9e3d5054d6f2d2a6: Deleting consensus metadata
I20260926 20:18:22.395694 13071 log.cc:1196] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/7b9b64766a5149779623fb8af746210b
I20260926 20:18:22.396122 13071 ts_tablet_manager.cc:1939] T 7b9b64766a5149779623fb8af746210b P 118c5e39e93f4ba78fa5a421ec93ae25: Deleting consensus metadata
I20260926 20:18:22.397738 12531 catalog_manager.cc:4841] TS de0cbc0b8a66400f9e3d5054d6f2d2a6 (127.31.119.194:37939): tablet 7b9b64766a5149779623fb8af746210b (table test-table [id=dce8e7b18c814cb1993ae9f716b92877]) successfully deleted
I20260926 20:18:22.398658 12534 catalog_manager.cc:4841] TS 118c5e39e93f4ba78fa5a421ec93ae25 (127.31.119.195:45057): tablet 7b9b64766a5149779623fb8af746210b (table test-table [id=dce8e7b18c814cb1993ae9f716b92877]) successfully deleted
W20260926 20:18:22.403558 12533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:40471 (error 108)
I20260926 20:18:22.404888 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 12749
I20260926 20:18:22.406905 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:22.406174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40471 (local address 127.0.0.1:47786)
0926 20:18:22.406717 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:22.406803 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437}
W20260926 20:18:22.407208 12533 catalog_manager.cc:4584] TS ac7ef72f52134e8f96b6c31e5e35e842 (127.31.119.193:40471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b9b64766a5149779623fb8af746210b: Network error: Client connection negotiation failed: client connection to 127.31.119.193:40471: connect: Connection refused (error 111)
I20260926 20:18:22.412663 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 12882
I20260926 20:18:22.423429 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 12514
I20260926 20:18:22.432518 32223 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:18:22.767732 32223 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:18:22.767890 32223 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:18:23.113809 32223 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:18:23.114 UTC [11581] LOG: received smart shutdown request
2026-09-26 20:18:23.118 UTC [11581] LOG: background worker "logical replication launcher" (PID 11590) exited with exit code 1
2026-09-26 20:18:23.118 UTC [11585] LOG: shutting down
2026-09-26 20:18:23.333 UTC [11581] LOG: database system is shut down
2026-09-26T20:18:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50029 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:18:23 dist-test-slave-vnbh krb5kdc[13083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:18:23 dist-test-slave-vnbh krb5kdc[13083](info): set up 2 sockets
Sep 26 20:18:23 dist-test-slave-vnbh krb5kdc[13083](info): commencing operation
krb5kdc: starting...
W20260926 20:18:24.522480 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:18:24 dist-test-slave-vnbh krb5kdc[13083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:18:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:18:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:18:24.657159 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:43091
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:41713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:43091,127.31.119.253:35923,127.31.119.252:43619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:24.771638 13096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:24.771981 13096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:24.772063 13096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:24.776283 13096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:18:24.776402 13096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:24.776441 13096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:24.776481 13096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:18:24.776515 13096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:18:24.781888 13096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:43091,127.31.119.253:35923,127.31.119.252:43619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:43091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:24.783012 13096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:24.784101 13096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:18:24.793879 13096 server_base.cc:1034] running on GCE node
W20260926 20:18:24.793819 13101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:24.793817 13104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:24.793856 13102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:24.794727 13096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:24.795884 13096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:24.797080 13096 hybrid_clock.cc:648] HybridClock initialized: now 1790453904797066 us; error 29 us; skew 500 ppm
Sep 26 20:18:24 dist-test-slave-vnbh krb5kdc[13083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453904, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:24.800055 13096 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:18:24.801071 13096 webserver.cc:460] Webserver started at http://127.31.119.254:37257/ using document root <none> and password file <none>
I20260926 20:18:24.801769 13096 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:24.801877 13096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:24.802090 13096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:24.804214 13096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "d3fb6cc589184b6c972bd892335a142e"
format_stamp: "Formatted at 2026-09-26 20:18:24 on dist-test-slave-vnbh"
server_key: "afd4dc42ea6f4881cc22dd7c4af9b764"
server_key_iv: "ef24ff74cf0fa24d0cd18d16813a59ee"
server_key_version: "encryptionkey@0"
I20260926 20:18:24.804834 13096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "d3fb6cc589184b6c972bd892335a142e"
format_stamp: "Formatted at 2026-09-26 20:18:24 on dist-test-slave-vnbh"
server_key: "afd4dc42ea6f4881cc22dd7c4af9b764"
server_key_iv: "ef24ff74cf0fa24d0cd18d16813a59ee"
server_key_version: "encryptionkey@0"
I20260926 20:18:24.808728 13096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:18:24.811275 13111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:24.812565 13096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:18:24.812732 13096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "d3fb6cc589184b6c972bd892335a142e"
format_stamp: "Formatted at 2026-09-26 20:18:24 on dist-test-slave-vnbh"
server_key: "afd4dc42ea6f4881cc22dd7c4af9b764"
server_key_iv: "ef24ff74cf0fa24d0cd18d16813a59ee"
server_key_version: "encryptionkey@0"
I20260926 20:18:24.812860 13096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:24.832481 13096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:24.835855 13096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:24.836076 13096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:24.843545 13096 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:43091
I20260926 20:18:24.844046 13163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:43091 every 8 connection(s)
I20260926 20:18:24.844604 13096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:18:24.847604 13164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:24.851406 13164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:18:24.854242 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 13096
I20260926 20:18:24.854449 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:18:24.854750 32223 external_mini_cluster.cc:1428] Setting key 85fef668c04562abe608f75660d39d4e
Sep 26 20:18:24 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453904, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.253@KRBTEST.COM" created.
I20260926 20:18:24.869096 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:24.854944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.31.119.254:43091)
0926 20:18:24.855309 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:24.855322 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:24.855387 (+ 65us) server_negotiation.cc:407] Connection header received
0926 20:18:24.855910 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:24.855919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:24.856183 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:24.856415 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:24.857742 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:24.858650 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:24.860403 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:24.860817 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:24.860969 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:24.863895 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:24.863901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:24.863913 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:24.863938 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:24.865731 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:24.866252 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:24.866257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:24.866258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:24.866370 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:24.866734 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:24.866737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:24.866739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:24.867394 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:24.867543 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:24.867953 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 20:18:24.868248 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
I20260926 20:18:24.869102 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:24.853934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:43091 (local address 127.0.0.1:52826)
0926 20:18:24.854483 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:24.854543 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:24.855564 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:24.856458 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:24.856467 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:24.857098 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:24.857555 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:24.857579 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:24.858798 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:24.858805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:24.860245 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:24.860261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:24.861015 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:24.861020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:24.861151 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:24.861811 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:24.861833 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:24.863718 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:24.865882 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:24.865893 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:24.865902 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:24.866138 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:24.866472 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:24.866477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:24.866479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:24.866631 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:24.867529 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:24.867535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:24.867668 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 20:18:24.867814 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 20:18:24.868109 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":159,"threads_started":1}
I20260926 20:18:24.872057 13164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:18:24.872980 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:24.872606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:35923 (local address 127.0.0.1:41938)
0926 20:18:24.872782 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:24.872872 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:35923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 20:18:24.873258 13115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.119.253:35923: connect: Connection refused (error 111)
I20260926 20:18:24.874128 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:24.873923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:35923 (local address 127.0.0.1:41948)
0926 20:18:24.874036 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:24.874078 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:35923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 20:18:24.874348 13164 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.253:35923: Network error: Client connection negotiation failed: client connection to 127.31.119.253:35923: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab.
Entry for principal kudu/127.31.119.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab.
I20260926 20:18:24.885206 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.253:35923
--webserver_interface=127.31.119.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab
--principal=kudu/127.31.119.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:41713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:43091,127.31.119.253:35923,127.31.119.252:43619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:18:24.923645 13164 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } attempt: 1
I20260926 20:18:24.924700 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:24.924393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:35923 (local address 127.0.0.1:41950)
0926 20:18:24.924557 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:24.924621 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:35923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 20:18:24.925554 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:24.925256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:35923 (local address 127.0.0.1:41958)
0926 20:18:24.925428 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:24.925484 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:35923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 20:18:24.925727 13164 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.253:35923: Network error: Client connection negotiation failed: client connection to 127.31.119.253:35923: connect: Connection refused (error 111)
I20260926 20:18:24.993920 13164 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } attempt: 2
I20260926 20:18:24.994853 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:24.994493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:35923 (local address 127.0.0.1:41970)
0926 20:18:24.994686 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:24.994759 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:35923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 20:18:24.995631 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:24.995401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:35923 (local address 127.0.0.1:41978)
0926 20:18:24.995507 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:24.995565 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:35923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:18:24.995786 13164 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.253:35923: Network error: Client connection negotiation failed: client connection to 127.31.119.253:35923: connect: Connection refused (error 111)
W20260926 20:18:24.999766 13171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:25.000078 13171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:25.000159 13171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:25.004444 13171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:18:25.004521 13171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:25.004544 13171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:25.004561 13171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:18:25.004606 13171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:18:25.010556 13171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:43091,127.31.119.253:35923,127.31.119.252:43619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab
--principal=kudu/127.31.119.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.253:35923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.119.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:25.011775 13171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:25.012867 13171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:25.022024 13179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:25.022306 13171 server_base.cc:1034] running on GCE node
W20260926 20:18:25.022043 13177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:25.022024 13176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:25.023100 13171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:25.024305 13171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:25.025476 13171 hybrid_clock.cc:648] HybridClock initialized: now 1790453905025439 us; error 52 us; skew 500 ppm
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453905, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:25.028555 13171 init.cc:377] Logged in from keytab as kudu/127.31.119.253@KRBTEST.COM (short username kudu)
I20260926 20:18:25.029584 13171 webserver.cc:460] Webserver started at http://127.31.119.253:40327/ using document root <none> and password file <none>
I20260926 20:18:25.030180 13171 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:25.030275 13171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:25.030524 13171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:25.032768 13171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/data/instance:
uuid: "03a70dcac5c543f8a9b027cf0e6aef73"
format_stamp: "Formatted at 2026-09-26 20:18:25 on dist-test-slave-vnbh"
server_key: "d821b1f9936a5983b6c18d2fdcf8dee4"
server_key_iv: "b759c17ef7a190a33ad8555594313a2f"
server_key_version: "encryptionkey@0"
I20260926 20:18:25.033561 13171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/wal/instance:
uuid: "03a70dcac5c543f8a9b027cf0e6aef73"
format_stamp: "Formatted at 2026-09-26 20:18:25 on dist-test-slave-vnbh"
server_key: "d821b1f9936a5983b6c18d2fdcf8dee4"
server_key_iv: "b759c17ef7a190a33ad8555594313a2f"
server_key_version: "encryptionkey@0"
I20260926 20:18:25.037811 13171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:18:25.040428 13186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:25.042064 13171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:25.042232 13171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/wal
uuid: "03a70dcac5c543f8a9b027cf0e6aef73"
format_stamp: "Formatted at 2026-09-26 20:18:25 on dist-test-slave-vnbh"
server_key: "d821b1f9936a5983b6c18d2fdcf8dee4"
server_key_iv: "b759c17ef7a190a33ad8555594313a2f"
server_key_version: "encryptionkey@0"
I20260926 20:18:25.042378 13171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:25.056581 13171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:25.059898 13171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:25.060108 13171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:25.067593 13171 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.253:35923
I20260926 20:18:25.067989 13238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.253:35923 every 8 connection(s)
I20260926 20:18:25.068665 13171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
I20260926 20:18:25.071647 13239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:25.071924 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 13171
I20260926 20:18:25.072082 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-1/wal/instance
I20260926 20:18:25.072402 32223 external_mini_cluster.cc:1428] Setting key f20b9bd3b94073a99ceba705f6d2f4ce
I20260926 20:18:25.076596 13239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.119.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.252@KRBTEST.COM" created.
I20260926 20:18:25.086989 13164 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } attempt: 3
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453905, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:25.093340 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.078725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.31.119.254:43091)
0926 20:18:25.078923 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.078930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.079797 (+ 867us) server_negotiation.cc:407] Connection header received
0926 20:18:25.080829 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.080833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.080892 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.080980 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.082362 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.082905 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.084709 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.085082 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.085214 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.088321 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.088327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.088332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.088365 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.090747 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.091404 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.091407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.091409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.091505 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.091944 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.091950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.091953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.092219 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.092306 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.093069 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.093168 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:18:25.094398 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.079097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:43091 (local address 127.0.0.1:52834)
0926 20:18:25.079612 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.079666 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:25.080593 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:25.081207 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:25.081218 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:25.081651 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:25.082159 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.082173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.083079 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.083082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.084550 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.084566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.085262 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.085267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.085496 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.086162 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:25.086178 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:25.088134 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:25.090868 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.090875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.090893 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.091291 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.091606 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.091610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.091612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.091805 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.092410 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:25.092421 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:25.092761 (+ 340us) client_negotiation.cc:769] Sending connection context
0926 20:18:25.093004 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 20:18:25.093359 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":149,"threads_started":1}
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453904, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
I20260926 20:18:25.095983 13239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:18:25.100035 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.087525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:35923 (local address 127.0.0.1:41988)
0926 20:18:25.087676 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.087694 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:25.087789 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:25.088549 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:25.088553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:25.088738 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:25.088889 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.088900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.090516 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.090520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.091187 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.091195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.091925 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.091928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.092043 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.092497 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:25.092517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:25.095852 (+ 3335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:25.098275 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.098279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.098281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.098533 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.098800 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.098802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.098803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.098983 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.099489 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:25.099492 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:25.099595 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 20:18:25.099715 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 20:18:25.099860 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:18:25.100119 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.087687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.31.119.253:35923)
0926 20:18:25.088116 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.088129 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.088155 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:18:25.088197 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.088205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.088418 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.088530 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.089043 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.090373 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.091303 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.091807 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.091933 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.096042 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.096050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.096062 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.096083 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.098143 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.098636 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.098641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.098642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.098717 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.099078 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.099081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.099083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.099370 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.099505 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.099824 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.099956 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab.
Entry for principal kudu/127.31.119.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab.
I20260926 20:18:25.101588 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.252:43619
--webserver_interface=127.31.119.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab
--principal=kudu/127.31.119.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:41713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:43091,127.31.119.253:35923,127.31.119.252:43619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453905, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
I20260926 20:18:25.102494 13164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:18:25.103345 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.103039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50680)
0926 20:18:25.103231 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.103284 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260926 20:18:25.104369 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.104062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50694)
0926 20:18:25.104240 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.104297 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260926 20:18:25.104532 13164 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:43619: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
I20260926 20:18:25.106853 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.096589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.31.119.253:35923)
0926 20:18:25.096936 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.096944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.096957 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:18:25.096993 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.096995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.097028 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.097441 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.097853 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.098531 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.099707 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.100121 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.100221 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.103227 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.103233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.103238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.103266 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.105061 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.105593 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.105596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.105597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.105668 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.106000 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.106003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.106005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.106208 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.106285 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.106611 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.106731 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":277,"thread_start_us":184,"threads_started":1}
I20260926 20:18:25.107085 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.096501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:35923 (local address 127.0.0.1:42004)
0926 20:18:25.096631 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.096644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:25.096718 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:25.097449 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:25.097452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:25.097670 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:25.097738 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.097743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.098801 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.098804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.099598 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.099606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.100346 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.100349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.100468 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.100971 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:25.100992 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:25.102539 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:25.105169 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.105175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.105180 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.105488 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.105761 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.105767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.105772 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.105875 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.106309 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:25.106314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:25.106434 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 20:18:25.106565 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 20:18:25.106808 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":63}
I20260926 20:18:25.108050 13239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:18:25.108767 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.108530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50710)
0926 20:18:25.108645 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.108716 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 20:18:25.108985 13188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
I20260926 20:18:25.109941 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.109731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50724)
0926 20:18:25.109837 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.109887 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:18:25.110098 13239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:43619: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
I20260926 20:18:25.135728 13164 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } attempt: 1
I20260926 20:18:25.136724 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.136359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50732)
0926 20:18:25.136548 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.136624 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 20:18:25.137761 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.137500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50744)
0926 20:18:25.137632 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.137689 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 20:18:25.137930 13164 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:43619: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
I20260926 20:18:25.159344 13239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } attempt: 1
I20260926 20:18:25.160293 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.159988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50756)
0926 20:18:25.160143 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.160210 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 20:18:25.161193 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.160940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50770)
0926 20:18:25.161074 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.161133 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 20:18:25.161350 13239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:43619: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
I20260926 20:18:25.213189 13164 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } attempt: 2
I20260926 20:18:25.214264 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.213860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50774)
0926 20:18:25.214107 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.214173 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 20:18:25.215036 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.214811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50782)
0926 20:18:25.214923 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.214977 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 20:18:25.215175 13164 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:43619: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
W20260926 20:18:25.218652 13246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:25.218952 13246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:25.219053 13246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:25.223204 13246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:18:25.223286 13246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:25.223335 13246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:25.223374 13246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:18:25.223461 13246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:18:25.229398 13246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:43091,127.31.119.253:35923,127.31.119.252:43619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab
--principal=kudu/127.31.119.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.252:43619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.119.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:25.229524 13239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } attempt: 2
I20260926 20:18:25.230322 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.230044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50794)
0926 20:18:25.230185 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.230249 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 20:18:25.230664 13246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:25.231113 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.230884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50806)
0926 20:18:25.231003 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.231051 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 20:18:25.231254 13239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:43619: Network error: Client connection negotiation failed: client connection to 127.31.119.252:43619: connect: Connection refused (error 111)
I20260926 20:18:25.231799 13246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:25.240453 13253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:25.240501 13252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:25.240453 13255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:25.240693 13246 server_base.cc:1034] running on GCE node
I20260926 20:18:25.241425 13246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:25.242601 13246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:25.243791 13246 hybrid_clock.cc:648] HybridClock initialized: now 1790453905243778 us; error 30 us; skew 500 ppm
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453905, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:25.246737 13246 init.cc:377] Logged in from keytab as kudu/127.31.119.252@KRBTEST.COM (short username kudu)
I20260926 20:18:25.247780 13246 webserver.cc:460] Webserver started at http://127.31.119.252:39909/ using document root <none> and password file <none>
I20260926 20:18:25.248317 13246 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:25.248374 13246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:25.248598 13246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:25.250669 13246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/data/instance:
uuid: "a03f2163a5944d0d9a66ccbe7b09cf61"
format_stamp: "Formatted at 2026-09-26 20:18:25 on dist-test-slave-vnbh"
server_key: "51abb86ad4b407b1365482372665541d"
server_key_iv: "813e27f4549ed049b66552391c5c9bd6"
server_key_version: "encryptionkey@0"
I20260926 20:18:25.251183 13246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/wal/instance:
uuid: "a03f2163a5944d0d9a66ccbe7b09cf61"
format_stamp: "Formatted at 2026-09-26 20:18:25 on dist-test-slave-vnbh"
server_key: "51abb86ad4b407b1365482372665541d"
server_key_iv: "813e27f4549ed049b66552391c5c9bd6"
server_key_version: "encryptionkey@0"
I20260926 20:18:25.254962 13246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:18:25.257567 13262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:25.258925 13246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:25.259024 13246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/wal
uuid: "a03f2163a5944d0d9a66ccbe7b09cf61"
format_stamp: "Formatted at 2026-09-26 20:18:25 on dist-test-slave-vnbh"
server_key: "51abb86ad4b407b1365482372665541d"
server_key_iv: "813e27f4549ed049b66552391c5c9bd6"
server_key_version: "encryptionkey@0"
I20260926 20:18:25.259114 13246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:25.277671 13246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:25.280879 13246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:25.281078 13246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:25.288318 13246 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.252:43619
I20260926 20:18:25.288757 13314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.252:43619 every 8 connection(s)
I20260926 20:18:25.289439 13246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
I20260926 20:18:25.292290 13315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:25.296096 13315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:18:25.299463 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 13246
I20260926 20:18:25.299618 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453737971229-32223-0/minicluster-data/master-2/wal/instance
I20260926 20:18:25.299897 32223 external_mini_cluster.cc:1428] Setting key 7b819240fe9e2d9b1c7ea81d0c4f7e37
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453905, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
I20260926 20:18:25.312109 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.298040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.31.119.254:43091)
0926 20:18:25.298212 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.298217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.299183 (+ 966us) server_negotiation.cc:407] Connection header received
0926 20:18:25.300267 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.300273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.300335 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.300420 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.301982 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.302624 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.304040 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.304368 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.304477 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.307772 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.307776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.307779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.307811 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.309529 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.310140 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.310142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.310144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.310218 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.310599 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.310602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.310604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.310891 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.310995 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.311655 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.311782 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:18:25.312820 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.298442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:43091 (local address 127.0.0.1:52844)
0926 20:18:25.299013 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.299057 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:25.300003 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:25.300642 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:25.300651 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:25.301172 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:25.301828 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.301843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.302745 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.302749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.303934 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.303941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.304538 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.304541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.304700 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.305849 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:25.305866 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:25.307599 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:25.309642 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.309651 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.309662 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.310040 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.310305 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.310311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.310315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.310488 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.311004 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:25.311013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:25.311297 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 20:18:25.311508 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 20:18:25.311854 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"spinlock_wait_cycles":768,"thread_start_us":114,"threads_started":1}
I20260926 20:18:25.312820 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.300684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.31.119.252:43619)
0926 20:18:25.301117 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.301130 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.301156 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:18:25.301198 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.301203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.301435 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.301601 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.302427 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.303754 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.304878 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.305426 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.305516 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.308228 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.308238 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.308255 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.308295 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.310249 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.310702 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.310708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.310710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.310797 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.311082 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.311085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.311087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.311355 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.311564 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.311797 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.311959 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":170,"threads_started":1}
I20260926 20:18:25.314253 13315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:18:25.314383 13164 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } attempt: 3
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453904, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453905, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:25.322435 13239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } attempt: 3
I20260926 20:18:25.326438 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.314907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50826)
0926 20:18:25.315147 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.315163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:25.315245 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:25.316211 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:25.316214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:25.316404 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:25.316523 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.316533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.317673 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.317676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.318430 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.318439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.319054 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.319061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.319168 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.319589 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:25.319601 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:25.321274 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:25.324323 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.324329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.324331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.324594 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.324998 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.325001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.325004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.325095 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.325858 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:25.325861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:25.325965 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 20:18:25.326046 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:18:25.326201 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":5,"spinlock_wait_cycles":8320}
I20260926 20:18:25.326721 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.315033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.31.119.253:35923)
0926 20:18:25.315146 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.315151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.315291 (+ 140us) server_negotiation.cc:407] Connection header received
0926 20:18:25.315478 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.315481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.315535 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.315632 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.316322 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.317197 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.318217 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.318468 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.318586 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.322701 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.322707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.322713 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.322745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.324618 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.325194 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.325200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.325201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.325352 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.325757 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.325763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.325768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.325980 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.326057 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.326415 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.326545 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:18:25.327280 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.314768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:35923 (local address 127.0.0.1:42010)
0926 20:18:25.315233 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.315245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:25.315361 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:25.315773 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:25.315776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:25.316028 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:25.316172 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.316181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.317416 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.317419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.318070 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.318079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.318582 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.318585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.318685 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.319413 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:25.319429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:25.322563 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:25.324746 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.324748 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.324751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.325070 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.325480 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.325486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.325491 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.325646 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.326129 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:25.326132 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:25.326222 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 20:18:25.326974 (+ 752us) client_negotiation.cc:240] Negotiation successful
0926 20:18:25.327119 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"spinlock_wait_cycles":163200}
I20260926 20:18:25.327775 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.315274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.31.119.252:43619)
0926 20:18:25.315978 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.315983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.315996 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:18:25.316030 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.316033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.316089 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.316631 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.316722 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.317552 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.318560 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.318938 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.319028 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.321473 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.321477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.321479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.321503 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.324189 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.324742 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.324745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.324747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.324876 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.325211 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.325214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.325215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.325735 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.327088 (+ 1353us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.327158 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.327250 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"spinlock_wait_cycles":15744}
I20260926 20:18:25.327869 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.315763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.31.119.254:43091)
0926 20:18:25.315866 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.315871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.315987 (+ 116us) server_negotiation.cc:407] Connection header received
0926 20:18:25.316239 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.316244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.316291 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.316374 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.317332 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.318176 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.319270 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.319487 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.319594 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.323057 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.323308 (+ 251us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.323312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.323339 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.325492 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.326010 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.326014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.326016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.326108 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.326408 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.326411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.326413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.326607 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.326862 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.327524 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.327642 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":1280}
I20260926 20:18:25.328235 13315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453905, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
I20260926 20:18:25.332178 13164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e: Bootstrap starting.
I20260926 20:18:25.333364 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.322867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50836)
0926 20:18:25.323081 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.323097 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:25.323182 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:25.323684 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:25.323688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:25.323880 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:25.323994 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.324000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.326066 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.326071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.326880 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.326894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.327533 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.327537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.327625 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.328080 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:25.328094 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:25.329638 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:25.331668 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.331671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.331673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.331907 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.332229 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.332232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.332234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.332334 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.332789 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:25.332803 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:25.332912 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 20:18:25.333037 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 20:18:25.333177 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 20:18:25.333408 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.322962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.31.119.252:43619)
0926 20:18:25.323451 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.323459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.323472 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:18:25.323511 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.323514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.323563 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.323657 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.324115 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.325921 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.327025 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.327402 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.329490 (+ 2088us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.329790 (+ 300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.329796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.329798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.329822 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.331546 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.332027 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.332030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.332032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.332112 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.332447 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.332450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.332452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.332672 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.332767 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.333101 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.333208 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":141,"threads_started":1}
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453905, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
I20260926 20:18:25.335402 13164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:25.336323 13164 log.cc:824] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:25.336710 13239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73: Bootstrap starting.
I20260926 20:18:25.337225 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.328708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:43619 (local address 127.0.0.1:50852)
0926 20:18:25.329082 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:25.329098 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:25.329174 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:25.329708 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:25.329710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:25.329875 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:25.329932 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.329936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.330942 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.330944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.331448 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:25.331455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.331898 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.331901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:25.331997 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.332426 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:25.332439 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:25.333856 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:25.335706 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.335711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.335717 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.335956 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.336257 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:25.336260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:25.336262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:25.336352 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:25.336764 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:25.336766 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:25.336863 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 20:18:25.336957 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:18:25.337076 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":1}
I20260926 20:18:25.337635 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.329415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.31.119.252:43619)
0926 20:18:25.329517 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.329521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.329532 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:18:25.329564 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.329567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.329599 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.329700 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.330034 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.330638 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.331561 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.331787 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.331901 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.333986 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.333989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.333992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.334012 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.335579 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.336064 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.336067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.336069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.336154 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.336452 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.336455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.336456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.336639 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.336727 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.337409 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.337508 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":45}
I20260926 20:18:25.338568 13164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e: No bootstrap required, opened a new log
I20260926 20:18:25.339509 13239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:25.340343 13239 log.cc:824] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:25.340516 13315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61: Bootstrap starting.
I20260926 20:18:25.341806 13164 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } }
I20260926 20:18:25.342069 13164 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:25.342132 13164 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3fb6cc589184b6c972bd892335a142e, State: Initialized, Role: FOLLOWER
I20260926 20:18:25.342583 13239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73: No bootstrap required, opened a new log
I20260926 20:18:25.342949 13164 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } }
I20260926 20:18:25.343096 13315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:25.343997 13315 log.cc:824] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:25.344465 13164 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:18:25.347651 13239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } }
I20260926 20:18:25.347950 13239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:25.348057 13239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a70dcac5c543f8a9b027cf0e6aef73, State: Initialized, Role: FOLLOWER
Sep 26 20:18:25 dist-test-slave-vnbh krb5kdc[13083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
I20260926 20:18:25.348871 13239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } }
W20260926 20:18:25.349735 13335 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:18:25.350287 13335 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:18:25.350366 13239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:18:25.350169 13323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } } }
I20260926 20:18:25.350446 13323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:18:25.350893 13315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61: No bootstrap required, opened a new log
I20260926 20:18:25.350188 13096 master_runner.cc:186] Error getting master registration for 127.31.119.253:35923: OK, instance_id { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" instance_seqno: 1790453905065341 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:18:25.354633 13096 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 20:18:25.355546 13348 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:18:25.355705 13348 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:18:25.356858 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.329036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42012 (local address 127.31.119.253:35923)
0926 20:18:25.329182 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.329186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.337078 (+ 7892us) server_negotiation.cc:407] Connection header received
0926 20:18:25.342660 (+ 5582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.342664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.342717 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.342807 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.343852 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.344613 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.345533 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.345823 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.345885 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.349012 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.349016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.349019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.349046 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.354616 (+ 5570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.355095 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.355098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.355100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.355186 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.355501 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.355504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.355506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.355728 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.355938 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.356585 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.356700 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 20:18:25.356891 13315 raft_consensus.cc:348] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } }
I20260926 20:18:25.357165 13315 raft_consensus.cc:374] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:25.357228 13315 raft_consensus.cc:729] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03f2163a5944d0d9a66ccbe7b09cf61, State: Initialized, Role: FOLLOWER
I20260926 20:18:25.357738 13337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } } }
I20260926 20:18:25.357931 13337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:18:25.357962 13315 consensus_queue.cc:260] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } }
I20260926 20:18:25.359414 13315 sys_catalog.cc:564] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:18:25.360865 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:18:25.363332 13350 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } } }
I20260926 20:18:25.363566 13350 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:18:25.365059 13171 master_runner.cc:263] No leader master found from master 03a70dcac5c543f8a9b027cf0e6aef73
I20260926 20:18:25.365131 13171 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:18:25.368273 13246 master_runner.cc:263] No leader master found from master a03f2163a5944d0d9a66ccbe7b09cf61
I20260926 20:18:25.368353 13246 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 20:18:25.370761 13361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:18:25.370930 13361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:18:25.386265 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.366175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.31.119.252:43619)
0926 20:18:25.366284 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.366289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.371629 (+ 5340us) server_negotiation.cc:407] Connection header received
0926 20:18:25.371795 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.371800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.371864 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.371917 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.372754 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.373579 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.376414 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.376765 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.376848 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.381492 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.381497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.381500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.381533 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.383188 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.384290 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.384294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.384296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.384397 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.384753 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.384757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.384758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.384992 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.385091 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.385953 (+ 862us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.386085 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:18:25.391285 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.367010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.31.119.253:35923)
0926 20:18:25.367115 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.367119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.373887 (+ 6768us) server_negotiation.cc:407] Connection header received
0926 20:18:25.375376 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.375380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.375431 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.375483 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.377418 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.378206 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.379395 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.379730 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.379801 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.385418 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.385422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.385425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.385456 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.387989 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.389749 (+ 1760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.389753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.389754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.389864 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.390173 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.390176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.390179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.390431 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.390564 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.390949 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.391090 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:18:25.395406 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:25.366627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.31.119.254:43091)
0926 20:18:25.366751 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:25.366756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:25.371283 (+ 4527us) server_negotiation.cc:407] Connection header received
0926 20:18:25.371434 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:25.371436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:25.371492 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:25.371570 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:25.375244 (+ 3674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.376022 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.389414 (+ 13392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:25.389757 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:25.389849 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:25.391563 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:25.391567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:25.391570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:25.391600 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:25.393326 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.393794 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.393797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.393799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.393880 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:25.394152 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:25.394156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:25.394157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:25.394372 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:25.394486 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:25.395115 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 20:18:25.395231 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:18:25.500617 13337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:18:25.500849 13337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } }
I20260926 20:18:25.502278 13337 leader_election.cc:288] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3fb6cc589184b6c972bd892335a142e (127.31.119.254:43091), a03f2163a5944d0d9a66ccbe7b09cf61 (127.31.119.252:43619)
I20260926 20:18:25.503070 13289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" is_pre_election: true
I20260926 20:18:25.503429 13289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a70dcac5c543f8a9b027cf0e6aef73 in term 0.
I20260926 20:18:25.504061 13188 leader_election.cc:302] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a70dcac5c543f8a9b027cf0e6aef73, a03f2163a5944d0d9a66ccbe7b09cf61; no voters:
I20260926 20:18:25.504901 13138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3fb6cc589184b6c972bd892335a142e" is_pre_election: true
I20260926 20:18:25.505227 13138 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a70dcac5c543f8a9b027cf0e6aef73 in term 0.
I20260926 20:18:25.506016 13337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:18:25.506103 13337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:18:25.506147 13337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:25.507555 13337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } }
I20260926 20:18:25.507936 13337 leader_election.cc:288] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [CANDIDATE]: Term 1 election: Requested vote from peers d3fb6cc589184b6c972bd892335a142e (127.31.119.254:43091), a03f2163a5944d0d9a66ccbe7b09cf61 (127.31.119.252:43619)
I20260926 20:18:25.508781 13289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61"
I20260926 20:18:25.508925 13289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:25.509486 13138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3fb6cc589184b6c972bd892335a142e"
I20260926 20:18:25.509611 13138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:25.510905 13138 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a70dcac5c543f8a9b027cf0e6aef73 in term 1.
I20260926 20:18:25.511328 13190 leader_election.cc:302] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a70dcac5c543f8a9b027cf0e6aef73, d3fb6cc589184b6c972bd892335a142e; no voters:
I20260926 20:18:25.512398 13289 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a70dcac5c543f8a9b027cf0e6aef73 in term 1.
I20260926 20:18:25.512945 13337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:18:25.513231 13337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [term 1 LEADER]: Becoming Leader. State: Replica: 03a70dcac5c543f8a9b027cf0e6aef73, State: Running, Role: LEADER
I20260926 20:18:25.513664 13337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } }
I20260926 20:18:25.515523 13337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03a70dcac5c543f8a9b027cf0e6aef73. Latest consensus state: current_term: 1 leader_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } } }
I20260926 20:18:25.515749 13337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:25.519528 13372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:18:25.520572 13372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:18:25.528461 13289 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [term 1 FOLLOWER]: Refusing update from remote peer 03a70dcac5c543f8a9b027cf0e6aef73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:18:25.529508 13370 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:18:25.532439 13268 tablet.cc:2369] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:18:25.534045 13350 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03a70dcac5c543f8a9b027cf0e6aef73. Latest consensus state: current_term: 1 leader_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } } }
I20260926 20:18:25.534176 13350 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:18:25.538164 13138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [term 1 FOLLOWER]: Refusing update from remote peer 03a70dcac5c543f8a9b027cf0e6aef73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:18:25.538892 13370 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:18:25.543329 13323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03a70dcac5c543f8a9b027cf0e6aef73. Latest consensus state: current_term: 1 leader_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } } }
I20260926 20:18:25.543524 13323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:18:25.546931 13370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } } }
I20260926 20:18:25.547083 13370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:25.548676 13323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } } }
I20260926 20:18:25.548807 13323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:18:25.549343 13373 mvcc.cc:205] Tried to move back new op lower bound from 7333699197035147264 to 7333699196987400192. Current Snapshot: MvccSnapshot[applied={T|T < 7333699197035147264}]
I20260926 20:18:25.549752 13376 mvcc.cc:205] Tried to move back new op lower bound from 7333699197035147264 to 7333699196987400192. Current Snapshot: MvccSnapshot[applied={T|T < 7333699197035147264}]
I20260926 20:18:25.549743 13337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } } }
I20260926 20:18:25.549860 13337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73 [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:25.551307 13350 sys_catalog.cc:455] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3fb6cc589184b6c972bd892335a142e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 43091 } } peers { permanent_uuid: "03a70dcac5c543f8a9b027cf0e6aef73" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 35923 } } peers { permanent_uuid: "a03f2163a5944d0d9a66ccbe7b09cf61" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 43619 } } }
I20260926 20:18:25.551465 13350 sys_catalog.cc:458] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:18:25.552541 13372 catalog_manager.cc:1347] Generated new cluster ID: 7d9c0703336b44c1a26020d3967a9834
I20260926 20:18:25.552628 13372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:18:25.554013 13374 mvcc.cc:205] Tried to move back new op lower bound from 7333699197035147264 to 7333699196987400192. Current Snapshot: MvccSnapshot[applied={T|T < 7333699197035147264}]
I20260926 20:18:25.576253 13372 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:18:25.577497 13372 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:18:25.584265 13372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03a70dcac5c543f8a9b027cf0e6aef73: Generated new TSK 0
I20260926 20:18:25.585251 13372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:18:26.351951 13335 catalog_manager.cc:1259] Loaded cluster ID: 7d9c0703336b44c1a26020d3967a9834
I20260926 20:18:26.352037 13335 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e: loading cluster ID for follower catalog manager: success
I20260926 20:18:26.353873 13335 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e: acquiring CA information for follower catalog manager: success
I20260926 20:18:26.355090 13335 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3fb6cc589184b6c972bd892335a142e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:18:26.372291 13361 catalog_manager.cc:1259] Loaded cluster ID: 7d9c0703336b44c1a26020d3967a9834
I20260926 20:18:26.372381 13361 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61: loading cluster ID for follower catalog manager: success
I20260926 20:18:26.374284 13361 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61: acquiring CA information for follower catalog manager: success
I20260926 20:18:26.375511 13361 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a03f2163a5944d0d9a66ccbe7b09cf61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:18:26.652267 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 13096
I20260926 20:18:26.658574 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 13171
I20260926 20:18:26.665122 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 13246
2026-09-26T20:18:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3203 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:18:26 dist-test-slave-vnbh krb5kdc[13387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:18:26 dist-test-slave-vnbh krb5kdc[13387](info): set up 2 sockets
Sep 26 20:18:26 dist-test-slave-vnbh krb5kdc[13387](info): commencing operation
krb5kdc: starting...
W20260926 20:18:27.718415 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:18:27 dist-test-slave-vnbh krb5kdc[13387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:18:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:18:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:18:27.845714 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:45219
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:45219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:27.956785 13400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:27.957083 13400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:27.957126 13400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:27.961975 13400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:18:27.962047 13400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:27.962069 13400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:27.962086 13400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:18:27.962102 13400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:18:27.967772 13400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:45219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:27.969156 13400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:27.970332 13400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:27.979302 13406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:27.979281 13408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:27.979259 13400 server_base.cc:1034] running on GCE node
W20260926 20:18:27.979220 13405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:27.980104 13400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:27.981395 13400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:27.982580 13400 hybrid_clock.cc:648] HybridClock initialized: now 1790453907982563 us; error 36 us; skew 500 ppm
Sep 26 20:18:27 dist-test-slave-vnbh krb5kdc[13387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453907, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:27.985582 13400 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:18:27.986577 13400 webserver.cc:460] Webserver started at http://127.31.119.254:41343/ using document root <none> and password file <none>
I20260926 20:18:27.987222 13400 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:27.987317 13400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:27.987568 13400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:27.989764 13400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "193d33d53eef46328a27ccfacd9396d4"
format_stamp: "Formatted at 2026-09-26 20:18:27 on dist-test-slave-vnbh"
server_key: "2804a66aaae63f4225bc656174627f09"
server_key_iv: "c603b87531c700fa0434525fbdb31051"
server_key_version: "encryptionkey@0"
I20260926 20:18:27.990356 13400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "193d33d53eef46328a27ccfacd9396d4"
format_stamp: "Formatted at 2026-09-26 20:18:27 on dist-test-slave-vnbh"
server_key: "2804a66aaae63f4225bc656174627f09"
server_key_iv: "c603b87531c700fa0434525fbdb31051"
server_key_version: "encryptionkey@0"
I20260926 20:18:27.994645 13400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:18:27.997486 13415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:27.998946 13400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:27.999119 13400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "193d33d53eef46328a27ccfacd9396d4"
format_stamp: "Formatted at 2026-09-26 20:18:27 on dist-test-slave-vnbh"
server_key: "2804a66aaae63f4225bc656174627f09"
server_key_iv: "c603b87531c700fa0434525fbdb31051"
server_key_version: "encryptionkey@0"
I20260926 20:18:27.999262 13400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:28.013607 13400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:28.017393 13400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:28.017612 13400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:28.024999 13400 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:45219
I20260926 20:18:28.025022 13467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:45219 every 8 connection(s)
I20260926 20:18:28.026052 13400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:18:28.028834 13468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:28.032384 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 13400
I20260926 20:18:28.032488 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:18:28.032804 32223 external_mini_cluster.cc:1428] Setting key 022e8c4080cc15680f964f4b5e485523
I20260926 20:18:28.035051 13468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4: Bootstrap starting.
I20260926 20:18:28.038100 13468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:28.039060 13468 log.cc:824] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:28.041712 13468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4: No bootstrap required, opened a new log
Sep 26 20:18:28 dist-test-slave-vnbh krb5kdc[13387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:28.046881 13468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193d33d53eef46328a27ccfacd9396d4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45219 } }
I20260926 20:18:28.047266 13468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:28.047355 13468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193d33d53eef46328a27ccfacd9396d4, State: Initialized, Role: FOLLOWER
I20260926 20:18:28.047955 13468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193d33d53eef46328a27ccfacd9396d4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45219 } }
I20260926 20:18:28.048128 13468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:18:28.048203 13468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:18:28.048328 13468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:28.049678 13468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193d33d53eef46328a27ccfacd9396d4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45219 } }
I20260926 20:18:28.050092 13468 leader_election.cc:302] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 193d33d53eef46328a27ccfacd9396d4; no voters:
I20260926 20:18:28.050400 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:28.034311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.31.119.254:45219)
0926 20:18:28.034767 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:28.034787 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:28.034833 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:18:28.035709 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:28.035734 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:28.036187 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:28.036615 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:28.037475 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:28.039052 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:28.040053 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:28.040750 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:28.040936 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:28.043938 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:28.043945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:28.043966 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:28.044008 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:28.046363 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:28.046942 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:28.046949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:28.046956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:28.047114 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:28.047577 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:28.047581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:28.047583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:28.048169 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:28.048392 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:28.048852 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 20:18:28.049212 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260926 20:18:28.050469 13468 leader_election.cc:288] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:18:28.050650 13473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:18:28.051074 13473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [term 1 LEADER]: Becoming Leader. State: Replica: 193d33d53eef46328a27ccfacd9396d4, State: Running, Role: LEADER
I20260926 20:18:28.051487 13473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193d33d53eef46328a27ccfacd9396d4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45219 } }
I20260926 20:18:28.052058 13468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:18:28.053579 13474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 193d33d53eef46328a27ccfacd9396d4. Latest consensus state: current_term: 1 leader_uuid: "193d33d53eef46328a27ccfacd9396d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193d33d53eef46328a27ccfacd9396d4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45219 } } }
I20260926 20:18:28.053714 13474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:28.054013 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:18:28.054034 13475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "193d33d53eef46328a27ccfacd9396d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193d33d53eef46328a27ccfacd9396d4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45219 } } }
I20260926 20:18:28.054137 13475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4 [sys.catalog]: This master's current role is: LEADER
W20260926 20:18:28.057276 13488 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:18:28.057461 13488 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:18:28.057540 13483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:18:28.058243 13483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:18:28.064139 13483 catalog_manager.cc:1347] Generated new cluster ID: 1631524a9ec44eb98cfeaf4de6053b79
I20260926 20:18:28.064224 13483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:18:28.098016 13483 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:18:28.099403 13483 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:18:28.108155 13483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 193d33d53eef46328a27ccfacd9396d4: Generated new TSK 0
I20260926 20:18:28.109015 13483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:18:28.161978 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 13400
2026-09-26T20:18:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1494 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:18:28 dist-test-slave-vnbh krb5kdc[13495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:18:28 dist-test-slave-vnbh krb5kdc[13495](info): set up 2 sockets
Sep 26 20:18:28 dist-test-slave-vnbh krb5kdc[13495](info): commencing operation
krb5kdc: starting...
W20260926 20:18:29.210537 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:18:29 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:18:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:18:29.337251 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38497
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:38497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:29.450194 13508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:29.450496 13508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:29.450538 13508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:29.454824 13508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:18:29.454890 13508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:29.454911 13508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:29.454928 13508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:18:29.454943 13508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:18:29.460354 13508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:38497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:29.461622 13508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:29.462746 13508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:29.471504 13514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:29.471522 13516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:29.471626 13513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:29.471653 13508 server_base.cc:1034] running on GCE node
I20260926 20:18:29.472406 13508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:29.473646 13508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:29.474833 13508 hybrid_clock.cc:648] HybridClock initialized: now 1790453909474800 us; error 49 us; skew 500 ppm
Sep 26 20:18:29 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453909, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:29.481982 13508 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:18:29.483031 13508 webserver.cc:460] Webserver started at http://127.31.119.254:33991/ using document root <none> and password file <none>
I20260926 20:18:29.483645 13508 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:29.483735 13508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:29.483982 13508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:29.486260 13508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "d00f1204703746c69cd6f343416750f2"
format_stamp: "Formatted at 2026-09-26 20:18:29 on dist-test-slave-vnbh"
server_key: "4be6caa1c1c995e566f328af06c51aca"
server_key_iv: "00af73ef91528402b84789f1ed3d6cbc"
server_key_version: "encryptionkey@0"
I20260926 20:18:29.486836 13508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "d00f1204703746c69cd6f343416750f2"
format_stamp: "Formatted at 2026-09-26 20:18:29 on dist-test-slave-vnbh"
server_key: "4be6caa1c1c995e566f328af06c51aca"
server_key_iv: "00af73ef91528402b84789f1ed3d6cbc"
server_key_version: "encryptionkey@0"
I20260926 20:18:29.490691 13508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:18:29.493377 13523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:29.494694 13508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:18:29.494843 13508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "d00f1204703746c69cd6f343416750f2"
format_stamp: "Formatted at 2026-09-26 20:18:29 on dist-test-slave-vnbh"
server_key: "4be6caa1c1c995e566f328af06c51aca"
server_key_iv: "00af73ef91528402b84789f1ed3d6cbc"
server_key_version: "encryptionkey@0"
I20260926 20:18:29.494997 13508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:29.527248 13508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:29.530706 13508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:29.530911 13508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:29.538383 13508 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:38497
I20260926 20:18:29.538714 13575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:38497 every 8 connection(s)
I20260926 20:18:29.539443 13508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:18:29.542382 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:29.544277 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 13508
I20260926 20:18:29.544452 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:18:29.544759 32223 external_mini_cluster.cc:1428] Setting key 61cce08bebe3bfcf4cd902852cef30e0
I20260926 20:18:29.548695 13576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Bootstrap starting.
I20260926 20:18:29.551499 13576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:29.552399 13576 log.cc:824] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:18:29 dist-test-slave-vnbh krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:29.554777 13576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: No bootstrap required, opened a new log
I20260926 20:18:29.558607 13576 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:29.558954 13576 raft_consensus.cc:374] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:29.559027 13576 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00f1204703746c69cd6f343416750f2, State: Initialized, Role: FOLLOWER
I20260926 20:18:29.559484 13576 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:29.559603 13576 raft_consensus.cc:388] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:18:29.559686 13576 raft_consensus.cc:482] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:18:29.559800 13576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:29.559962 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:29.546338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.31.119.254:38497)
0926 20:18:29.546756 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:29.546771 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:29.546806 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:18:29.547655 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:29.547672 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:29.548004 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:29.548285 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:29.549195 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:29.550257 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:29.551461 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:29.552023 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:29.552155 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:29.554929 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:29.554934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:29.554951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:29.554978 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:29.556718 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:29.557187 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:29.557193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:29.557197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:29.557389 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:29.557679 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:29.557684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:29.557685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:29.558130 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:29.558345 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:29.558605 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 20:18:29.558934 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":138,"threads_started":1}
I20260926 20:18:29.560835 13576 raft_consensus.cc:504] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:29.561189 13576 leader_election.cc:302] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d00f1204703746c69cd6f343416750f2; no voters:
I20260926 20:18:29.561565 13576 leader_election.cc:288] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:18:29.561645 13581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:18:29.561846 13581 raft_consensus.cc:686] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 1 LEADER]: Becoming Leader. State: Replica: d00f1204703746c69cd6f343416750f2, State: Running, Role: LEADER
I20260926 20:18:29.562201 13581 consensus_queue.cc:237] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:29.562929 13576 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:18:29.563903 13582 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00f1204703746c69cd6f343416750f2. Latest consensus state: current_term: 1 leader_uuid: "d00f1204703746c69cd6f343416750f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } } }
I20260926 20:18:29.564200 13583 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d00f1204703746c69cd6f343416750f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } } }
I20260926 20:18:29.564312 13582 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:29.564421 13583 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: This master's current role is: LEADER
W20260926 20:18:29.567626 13596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:18:29.567703 13596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:18:29.567786 13592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:18:29.568445 13592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:18:29.573783 13592 catalog_manager.cc:1347] Generated new cluster ID: 1c37ec29392245fca3efc25bf77ce73a
I20260926 20:18:29.573854 13592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:18:29.587584 13592 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:18:29.588534 13592 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:18:29.596625 13592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Generated new TSK 0
I20260926 20:18:29.597543 13592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:18:29.628872 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38497
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:29.749092 13602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:29.749468 13602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:29.749528 13602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:29.753677 13602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:29.753746 13602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:29.753813 13602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:18:29.759410 13602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:29.760535 13602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:29.761597 13602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:29.772100 13610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:29.772125 13607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:29.772125 13608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:29.772441 13602 server_base.cc:1034] running on GCE node
I20260926 20:18:29.772871 13602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:29.773629 13602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:29.774825 13602 hybrid_clock.cc:648] HybridClock initialized: now 1790453909774797 us; error 39 us; skew 500 ppm
Sep 26 20:18:29 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453909, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:29.778007 13602 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:18:29.778877 13602 webserver.cc:460] Webserver started at http://127.31.119.193:43875/ using document root <none> and password file <none>
I20260926 20:18:29.779450 13602 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:29.779542 13602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:29.779778 13602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:29.781965 13602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "a5f85a54b3d5432f8c2089249cc619a7"
format_stamp: "Formatted at 2026-09-26 20:18:29 on dist-test-slave-vnbh"
server_key: "5c8db35f4bcbb3f1bc556710d0eefc88"
server_key_iv: "c24067995d0fd81671c5973d6b352083"
server_key_version: "encryptionkey@0"
I20260926 20:18:29.782572 13602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "a5f85a54b3d5432f8c2089249cc619a7"
format_stamp: "Formatted at 2026-09-26 20:18:29 on dist-test-slave-vnbh"
server_key: "5c8db35f4bcbb3f1bc556710d0eefc88"
server_key_iv: "c24067995d0fd81671c5973d6b352083"
server_key_version: "encryptionkey@0"
I20260926 20:18:29.786418 13602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:18:29.788774 13617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:29.790066 13602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:18:29.790206 13602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "a5f85a54b3d5432f8c2089249cc619a7"
format_stamp: "Formatted at 2026-09-26 20:18:29 on dist-test-slave-vnbh"
server_key: "5c8db35f4bcbb3f1bc556710d0eefc88"
server_key_iv: "c24067995d0fd81671c5973d6b352083"
server_key_version: "encryptionkey@0"
I20260926 20:18:29.790316 13602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:29.819792 13602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:29.823575 13602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:29.823797 13602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:29.824429 13602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:29.825327 13602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:29.825397 13602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:29.825481 13602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:29.825525 13602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:29.833968 13602 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:44169
I20260926 20:18:29.833992 13730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:44169 every 8 connection(s)
I20260926 20:18:29.834867 13602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:18:29.837563 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 13602
I20260926 20:18:29.837699 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:18:29.838011 32223 external_mini_cluster.cc:1428] Setting key 76a7997561e199db967f4d3afac4d6a2
Sep 26 20:18:29 dist-test-slave-vnbh krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453909, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
I20260926 20:18:29.850481 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:29.836965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:39531 (local address 127.31.119.254:38497)
0926 20:18:29.837110 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:29.837114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:29.838248 (+ 1134us) server_negotiation.cc:407] Connection header received
0926 20:18:29.839869 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:29.839872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:29.839924 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:29.840048 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:29.841358 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:29.841936 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:29.843189 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:29.843434 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:29.843547 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:29.846484 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:29.846487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:29.846489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:29.846515 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:29.848002 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:29.848595 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:29.848600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:29.848602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:29.848686 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:29.849046 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:29.849050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:29.849052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:29.849259 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:29.849540 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:29.850155 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 20:18:29.850291 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:18:29.851328 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:29.837489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:39531)
0926 20:18:29.838068 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:29.838124 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:29.839656 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:29.840252 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:29.840259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:29.840645 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:29.841143 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:29.841158 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:29.842079 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:29.842083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:29.843054 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:29.843064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:29.843607 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:29.843612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:29.843792 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:29.844468 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:29.844485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:29.846289 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:29.848127 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:29.848134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:29.848151 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:29.848489 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:29.848782 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:29.848786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:29.848788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:29.848948 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:29.849547 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:29.849557 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:29.849832 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 20:18:29.850039 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 20:18:29.850414 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":205,"threads_started":1}
I20260926 20:18:29.852623 13731 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38497
I20260926 20:18:29.852895 13731 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:29.853480 13731 heartbeater.cc:511] Master 127.31.119.254:38497 requested a full tablet report, sending...
I20260926 20:18:29.855268 13540 ts_manager.cc:194] Registered new tserver with Master: a5f85a54b3d5432f8c2089249cc619a7 (127.31.119.193:44169)
I20260926 20:18:29.856997 13540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:39531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:18:29.868662 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38497
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:29.980196 13736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:29.980470 13736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:29.980515 13736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:29.984669 13736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:29.984748 13736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:29.984854 13736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:18:29.990592 13736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:29.991778 13736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:29.992847 13736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:30.003822 13742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:30.003922 13744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:30.003841 13741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:30.004261 13736 server_base.cc:1034] running on GCE node
I20260926 20:18:30.004804 13736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:30.005710 13736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:30.006943 13736 hybrid_clock.cc:648] HybridClock initialized: now 1790453910006903 us; error 63 us; skew 500 ppm
Sep 26 20:18:30 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453910, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:30.010035 13736 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:18:30.011032 13736 webserver.cc:460] Webserver started at http://127.31.119.194:37761/ using document root <none> and password file <none>
I20260926 20:18:30.011597 13736 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:30.011658 13736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:30.011839 13736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:30.014053 13736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "d08241111ed74731884a8c87ebcc0ed5"
format_stamp: "Formatted at 2026-09-26 20:18:30 on dist-test-slave-vnbh"
server_key: "e3720e159899c9520723260cdb4dedad"
server_key_iv: "928e1c10241662d70a8d1e081f393f12"
server_key_version: "encryptionkey@0"
I20260926 20:18:30.014612 13736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "d08241111ed74731884a8c87ebcc0ed5"
format_stamp: "Formatted at 2026-09-26 20:18:30 on dist-test-slave-vnbh"
server_key: "e3720e159899c9520723260cdb4dedad"
server_key_iv: "928e1c10241662d70a8d1e081f393f12"
server_key_version: "encryptionkey@0"
I20260926 20:18:30.018711 13736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:18:30.021507 13751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:30.023051 13736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:30.023218 13736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "d08241111ed74731884a8c87ebcc0ed5"
format_stamp: "Formatted at 2026-09-26 20:18:30 on dist-test-slave-vnbh"
server_key: "e3720e159899c9520723260cdb4dedad"
server_key_iv: "928e1c10241662d70a8d1e081f393f12"
server_key_version: "encryptionkey@0"
I20260926 20:18:30.023366 13736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:30.037835 13736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:30.041504 13736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:30.041728 13736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:30.042372 13736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:30.043234 13736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:30.043282 13736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:30.043322 13736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:30.043372 13736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:30.052873 13736 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:33057
I20260926 20:18:30.052973 13864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:33057 every 8 connection(s)
I20260926 20:18:30.054008 13736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:18:30.056128 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 13736
I20260926 20:18:30.056253 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:18:30.056578 32223 external_mini_cluster.cc:1428] Setting key c958243fb2b3e3782d090c26f167c787
Sep 26 20:18:30 dist-test-slave-vnbh krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453910, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
I20260926 20:18:30.070027 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.056037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:53937 (local address 127.31.119.254:38497)
0926 20:18:30.056172 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:30.056178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:30.057539 (+ 1361us) server_negotiation.cc:407] Connection header received
0926 20:18:30.059088 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:30.059094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:30.059159 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:30.059253 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:30.060854 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.061569 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.062733 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.062981 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.063101 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.065845 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:30.065850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:30.065854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:30.065886 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:30.067744 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:30.068325 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:30.068327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:30.068329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:30.068494 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:30.068845 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:30.068847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:30.068848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:30.069044 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:30.069122 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:30.069758 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 20:18:30.069864 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
I20260926 20:18:30.070972 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.056591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:53937)
0926 20:18:30.057159 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:30.057219 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:30.058773 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:30.059440 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:30.059451 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:30.059933 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:30.060687 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.060709 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.061686 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.061689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.062594 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.062608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.063152 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.063157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.063340 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.063987 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:30.064004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:30.065660 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:30.067887 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:30.067892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:30.067906 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:30.068182 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:30.068589 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:30.068595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:30.068600 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:30.068761 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:30.069139 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:30.069144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:30.069527 (+ 383us) client_negotiation.cc:769] Sending connection context
0926 20:18:30.069707 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 20:18:30.070005 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":150,"threads_started":1}
I20260926 20:18:30.072345 13865 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38497
I20260926 20:18:30.072628 13865 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:30.073240 13865 heartbeater.cc:511] Master 127.31.119.254:38497 requested a full tablet report, sending...
I20260926 20:18:30.074494 13540 ts_manager.cc:194] Registered new tserver with Master: d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057)
I20260926 20:18:30.075363 13540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:53937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:18:30.090076 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38497
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:30.205379 13870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:30.205672 13870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:30.205719 13870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:30.209873 13870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:30.209949 13870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:30.210012 13870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:18:30.215682 13870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:30.216905 13870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:30.218017 13870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:30.227862 13876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:30.227957 13878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:30.227946 13870 server_base.cc:1034] running on GCE node
W20260926 20:18:30.227856 13875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:30.228629 13870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:30.229234 13870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:30.230448 13870 hybrid_clock.cc:648] HybridClock initialized: now 1790453910230431 us; error 26 us; skew 500 ppm
Sep 26 20:18:30 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453910, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:30.233439 13870 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:18:30.234349 13870 webserver.cc:460] Webserver started at http://127.31.119.195:37551/ using document root <none> and password file <none>
I20260926 20:18:30.234918 13870 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:30.235016 13870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:30.235245 13870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:30.237341 13870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "43e93296fd254148a3023e132d7ee77a"
format_stamp: "Formatted at 2026-09-26 20:18:30 on dist-test-slave-vnbh"
server_key: "9fc8bfd6f0127c3829872e9fab8be4cd"
server_key_iv: "538a1f488d31eb365bcfc21d285a70a9"
server_key_version: "encryptionkey@0"
I20260926 20:18:30.237911 13870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "43e93296fd254148a3023e132d7ee77a"
format_stamp: "Formatted at 2026-09-26 20:18:30 on dist-test-slave-vnbh"
server_key: "9fc8bfd6f0127c3829872e9fab8be4cd"
server_key_iv: "538a1f488d31eb365bcfc21d285a70a9"
server_key_version: "encryptionkey@0"
I20260926 20:18:30.241657 13870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:18:30.244181 13885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:30.245522 13870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:18:30.245658 13870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "43e93296fd254148a3023e132d7ee77a"
format_stamp: "Formatted at 2026-09-26 20:18:30 on dist-test-slave-vnbh"
server_key: "9fc8bfd6f0127c3829872e9fab8be4cd"
server_key_iv: "538a1f488d31eb365bcfc21d285a70a9"
server_key_version: "encryptionkey@0"
I20260926 20:18:30.245779 13870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:30.260952 13870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:30.264464 13870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:30.264673 13870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:30.265365 13870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:30.266213 13870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:30.266284 13870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:30.266360 13870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:30.266413 13870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:30.275272 13870 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:46403
I20260926 20:18:30.275295 13998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:46403 every 8 connection(s)
I20260926 20:18:30.276182 13870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:18:30.277323 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 13870
I20260926 20:18:30.277459 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:18:30.277734 32223 external_mini_cluster.cc:1428] Setting key b5e295fcda38561203ad04b581a1cee7
Sep 26 20:18:30 dist-test-slave-vnbh krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453910, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:30.290369 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.278281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:41845 (local address 127.31.119.254:38497)
0926 20:18:30.278423 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:30.278428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:30.279610 (+ 1182us) server_negotiation.cc:407] Connection header received
0926 20:18:30.281029 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:30.281031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:30.281077 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:30.281144 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:30.282405 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.282963 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.284107 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.284369 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.284461 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.287181 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:30.287185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:30.287187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:30.287213 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:30.288473 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:30.288986 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:30.288989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:30.288990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:30.289067 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:30.289413 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:30.289415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:30.289416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:30.289592 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:30.289667 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:30.290147 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 20:18:30.290245 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:18:30.291425 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.278825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:41845)
0926 20:18:30.279412 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:30.279495 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:30.280832 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:30.281332 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:30.281340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:30.281710 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:30.282270 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.282286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.283069 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.283072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.283959 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.283968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.284517 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.284520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.284658 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.285217 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:30.285233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:30.286863 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:30.288573 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:30.288578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:30.288590 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:30.288838 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:30.289148 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:30.289150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:30.289151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:30.289279 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:30.289679 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:30.289685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:30.289936 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 20:18:30.290087 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 20:18:30.290469 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":148,"threads_started":1}
I20260926 20:18:30.292502 13999 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38497
I20260926 20:18:30.292764 13999 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:30.293257 13999 heartbeater.cc:511] Master 127.31.119.254:38497 requested a full tablet report, sending...
I20260926 20:18:30.294276 13540 ts_manager.cc:194] Registered new tserver with Master: 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403)
I20260926 20:18:30.294853 13540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:41845
I20260926 20:18:30.304106 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:18:30.313781 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.306201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.31.119.254:38497)
0926 20:18:30.306361 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:30.306367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:30.306572 (+ 205us) server_negotiation.cc:407] Connection header received
0926 20:18:30.306729 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:30.306732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:30.306786 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:30.306872 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:30.307611 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.308162 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.309016 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.309249 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.309402 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.310786 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:30.310789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:30.310792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:30.310818 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:30.312027 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:30.312450 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:30.312455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:30.312457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:30.312541 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:30.312811 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:30.312815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:30.312817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:30.313057 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:30.313198 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:30.313500 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 20:18:30.313620 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:18:30.316807 13540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:18:30.320072 13540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:18:30.336135 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.329559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:44169 (local address 127.0.0.1:36444)
0926 20:18:30.330149 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:30.330211 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:30.330365 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:30.331274 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:30.331281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:30.331323 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:30.331466 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.331473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.332704 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.332710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.334220 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.334232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.335563 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.335566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.335674 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.335712 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:18:30.335824 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:18:30.335963 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":1,"spinlock_wait_cycles":2560,"thread_start_us":107,"threads_started":1}
I20260926 20:18:30.336869 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.330196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:46403 (local address 127.0.0.1:55756)
0926 20:18:30.331131 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:30.331145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:30.331241 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:30.331967 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:30.331970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:30.331987 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:30.332066 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.332071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.333657 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.333660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.334880 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.334895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.336444 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.336450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.336553 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.336578 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:18:30.336653 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:18:30.336723 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":110,"threads_started":1}
I20260926 20:18:30.337347 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.330389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.31.119.195:46403)
0926 20:18:30.330818 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:30.330829 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:30.331349 (+ 520us) server_negotiation.cc:407] Connection header received
0926 20:18:30.331446 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:30.331450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:30.331651 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:30.331796 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:30.332200 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.333352 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.335068 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.336110 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.336230 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.336374 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:30.337060 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 20:18:30.337161 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":119,"threads_started":1}
I20260926 20:18:30.337567 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.330314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:33057 (local address 127.0.0.1:45686)
0926 20:18:30.331211 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:30.331223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:30.331322 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:30.331766 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:30.331769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:30.331787 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:30.331901 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.331912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.334263 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.334269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.335409 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.335418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.337157 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.337161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.337247 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.337275 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:18:30.337365 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:18:30.337433 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"spinlock_wait_cycles":4480,"thread_start_us":60,"threads_started":1}
I20260926 20:18:30.337606 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.329746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.31.119.193:44169)
0926 20:18:30.330036 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:30.330064 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:30.330538 (+ 474us) server_negotiation.cc:407] Connection header received
0926 20:18:30.330633 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:30.330639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:30.330875 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:30.331054 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:30.331596 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.332569 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.334358 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.335237 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.335354 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.335505 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:30.336800 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0926 20:18:30.336910 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":106,"threads_started":1}
I20260926 20:18:30.338002 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.330686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.31.119.194:33057)
0926 20:18:30.331073 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:30.331085 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:30.331304 (+ 219us) server_negotiation.cc:407] Connection header received
0926 20:18:30.331423 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:30.331428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:30.331620 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:30.331784 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:30.331996 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.333777 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.335553 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.336358 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.336481 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.336607 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:30.337751 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0926 20:18:30.337854 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":93,"threads_started":1}
I20260926 20:18:30.340497 13799 tablet_service.cc:1467] Processing CreateTablet for tablet e0da5de3bac54a4abf463be8650e9c29 (DEFAULT_TABLE table=test-table [id=083c2d13a2fb4c31af424b6f2b12c024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:30.340567 13665 tablet_service.cc:1467] Processing CreateTablet for tablet e0da5de3bac54a4abf463be8650e9c29 (DEFAULT_TABLE table=test-table [id=083c2d13a2fb4c31af424b6f2b12c024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:30.340569 13933 tablet_service.cc:1467] Processing CreateTablet for tablet e0da5de3bac54a4abf463be8650e9c29 (DEFAULT_TABLE table=test-table [id=083c2d13a2fb4c31af424b6f2b12c024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:30.341655 13665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0da5de3bac54a4abf463be8650e9c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:30.341660 13933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0da5de3bac54a4abf463be8650e9c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:30.341658 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0da5de3bac54a4abf463be8650e9c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:30.348248 14016 tablet_bootstrap.cc:492] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: Bootstrap starting.
I20260926 20:18:30.349900 14017 tablet_bootstrap.cc:492] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Bootstrap starting.
I20260926 20:18:30.351356 14016 tablet_bootstrap.cc:654] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:30.352248 14017 tablet_bootstrap.cc:654] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:30.352308 14016 log.cc:824] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:30.353107 14017 log.cc:824] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:30.353531 14018 tablet_bootstrap.cc:492] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: Bootstrap starting.
I20260926 20:18:30.355005 14017 tablet_bootstrap.cc:492] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: No bootstrap required, opened a new log
I20260926 20:18:30.355175 14016 tablet_bootstrap.cc:492] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: No bootstrap required, opened a new log
I20260926 20:18:30.355253 14017 ts_tablet_manager.cc:1397] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:18:30.355369 14016 ts_tablet_manager.cc:1397] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 20:18:30.355913 14018 tablet_bootstrap.cc:654] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:30.357250 14018 log.cc:824] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:30.358942 14017 raft_consensus.cc:348] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:30.359202 14017 raft_consensus.cc:374] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:30.359285 14017 raft_consensus.cc:729] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f85a54b3d5432f8c2089249cc619a7, State: Initialized, Role: FOLLOWER
I20260926 20:18:30.359372 14016 raft_consensus.cc:348] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:30.359668 14016 raft_consensus.cc:374] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:30.359691 14018 tablet_bootstrap.cc:492] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: No bootstrap required, opened a new log
I20260926 20:18:30.359758 14016 raft_consensus.cc:729] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08241111ed74731884a8c87ebcc0ed5, State: Initialized, Role: FOLLOWER
I20260926 20:18:30.359800 14017 consensus_queue.cc:260] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:30.359899 14018 ts_tablet_manager.cc:1397] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260926 20:18:30.360229 14016 consensus_queue.cc:260] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:30.360615 13731 heartbeater.cc:503] Master 127.31.119.254:38497 was elected leader, sending a full tablet report...
I20260926 20:18:30.361454 14016 ts_tablet_manager.cc:1428] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260926 20:18:30.361876 14017 ts_tablet_manager.cc:1428] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:18:30.361959 13865 heartbeater.cc:503] Master 127.31.119.254:38497 was elected leader, sending a full tablet report...
I20260926 20:18:30.364689 14018 raft_consensus.cc:348] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:30.364975 14018 raft_consensus.cc:374] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:30.365041 14018 raft_consensus.cc:729] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e93296fd254148a3023e132d7ee77a, State: Initialized, Role: FOLLOWER
I20260926 20:18:30.365556 14018 consensus_queue.cc:260] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:30.366231 13999 heartbeater.cc:503] Master 127.31.119.254:38497 was elected leader, sending a full tablet report...
I20260926 20:18:30.366607 14018 ts_tablet_manager.cc:1428] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260926 20:18:30.527362 14000 tablet.cc:2369] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:18:30.555248 13866 tablet.cc:2369] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:18:30.577529 14024 raft_consensus.cc:482] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:18:30.577773 14024 raft_consensus.cc:504] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:30.578972 14024 leader_election.cc:288] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f85a54b3d5432f8c2089249cc619a7 (127.31.119.193:44169), d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057)
I20260926 20:18:30.583890 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.579670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:47097 (local address 127.31.119.194:33057)
0926 20:18:30.579814 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:30.579818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:30.579990 (+ 172us) server_negotiation.cc:407] Connection header received
0926 20:18:30.580079 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:30.580081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:30.580147 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:30.580227 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:30.580449 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.581134 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.582614 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.583228 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.583358 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.583441 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:30.583684 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 20:18:30.583756 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:18:30.583897 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.579319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:37639 (local address 127.31.119.193:44169)
0926 20:18:30.579476 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:30.579481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:30.579499 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:18:30.579649 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:30.579652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:30.579694 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:30.579788 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:30.580139 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.580958 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.582614 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.583236 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.583358 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.583416 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:30.583684 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 20:18:30.583756 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:18:30.583911 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.579213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:44169 (local address 127.31.119.195:37639)
0926 20:18:30.579362 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:30.579380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:30.579511 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:30.579865 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:30.579869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:30.579889 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:30.580034 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.580042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.581064 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.581066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.582472 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.582483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.583361 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.583365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.583574 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.583590 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:18:30.583655 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:18:30.583751 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":87}
I20260926 20:18:30.583920 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.579523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:33057 (local address 127.31.119.195:47097)
0926 20:18:30.579885 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:30.579898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:30.579999 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:30.580240 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:30.580242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:30.580257 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:30.580343 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.580347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.581223 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.581226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.582472 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:30.582483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.583361 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.583365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:30.583546 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.583573 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:18:30.583655 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:18:30.583751 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":119,"thread_start_us":141,"threads_started":1}
I20260926 20:18:30.584617 13685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0da5de3bac54a4abf463be8650e9c29" candidate_uuid: "43e93296fd254148a3023e132d7ee77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f85a54b3d5432f8c2089249cc619a7" is_pre_election: true
I20260926 20:18:30.584805 13819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0da5de3bac54a4abf463be8650e9c29" candidate_uuid: "43e93296fd254148a3023e132d7ee77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08241111ed74731884a8c87ebcc0ed5" is_pre_election: true
I20260926 20:18:30.585013 13685 raft_consensus.cc:2454] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e93296fd254148a3023e132d7ee77a in term 0.
I20260926 20:18:30.585114 13819 raft_consensus.cc:2454] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e93296fd254148a3023e132d7ee77a in term 0.
I20260926 20:18:30.585520 13886 leader_election.cc:302] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e93296fd254148a3023e132d7ee77a, a5f85a54b3d5432f8c2089249cc619a7; no voters:
I20260926 20:18:30.585803 14024 raft_consensus.cc:2781] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:18:30.585899 14024 raft_consensus.cc:482] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:18:30.585966 14024 raft_consensus.cc:3037] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Advancing to term 1
W20260926 20:18:30.586226 13732 tablet.cc:2369] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:18:30.587112 14024 raft_consensus.cc:504] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:30.587471 14024 leader_election.cc:288] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [CANDIDATE]: Term 1 election: Requested vote from peers a5f85a54b3d5432f8c2089249cc619a7 (127.31.119.193:44169), d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057)
I20260926 20:18:30.587987 13685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0da5de3bac54a4abf463be8650e9c29" candidate_uuid: "43e93296fd254148a3023e132d7ee77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f85a54b3d5432f8c2089249cc619a7"
I20260926 20:18:30.588016 13819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0da5de3bac54a4abf463be8650e9c29" candidate_uuid: "43e93296fd254148a3023e132d7ee77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08241111ed74731884a8c87ebcc0ed5"
I20260926 20:18:30.588148 13819 raft_consensus.cc:3037] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:30.588151 13685 raft_consensus.cc:3037] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:30.589221 13819 raft_consensus.cc:2454] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e93296fd254148a3023e132d7ee77a in term 1.
I20260926 20:18:30.589258 13685 raft_consensus.cc:2454] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e93296fd254148a3023e132d7ee77a in term 1.
I20260926 20:18:30.589605 13886 leader_election.cc:302] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e93296fd254148a3023e132d7ee77a, a5f85a54b3d5432f8c2089249cc619a7; no voters:
I20260926 20:18:30.589816 14024 raft_consensus.cc:2781] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:18:30.590056 14024 raft_consensus.cc:686] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 1 LEADER]: Becoming Leader. State: Replica: 43e93296fd254148a3023e132d7ee77a, State: Running, Role: LEADER
I20260926 20:18:30.590404 14024 consensus_queue.cc:237] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:30.594177 13540 catalog_manager.cc:5495] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e93296fd254148a3023e132d7ee77a (127.31.119.195). New cstate: current_term: 1 leader_uuid: "43e93296fd254148a3023e132d7ee77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } health_report { overall_health: HEALTHY } } }
I20260926 20:18:30.632316 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.628108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.31.119.195:46403)
0926 20:18:30.628274 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:30.628279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:30.628295 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:18:30.628508 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:30.628511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:30.628573 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:30.628693 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:18:30.628961 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.629626 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.630636 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:30.630886 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:30.631007 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:30.631373 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:18:30.631856 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:18:30.632009 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:30.632054 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 20:18:30.632119 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:18:30.639328 13685 raft_consensus.cc:1261] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Refusing update from remote peer 43e93296fd254148a3023e132d7ee77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:18:30.639357 13819 raft_consensus.cc:1261] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5 [term 1 FOLLOWER]: Refusing update from remote peer 43e93296fd254148a3023e132d7ee77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:18:30.640472 14024 consensus_queue.cc:1035] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:18:30.640727 14026 consensus_queue.cc:1035] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:18:30.650852 14031 mvcc.cc:205] Tried to move back new op lower bound from 7333699217969352704 to 7333699217781018624. Current Snapshot: MvccSnapshot[applied={T|T < 7333699217969352704}]
I20260926 20:18:30.658499 13540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:18:30.658699 13540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:18:30.661193 13540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Sending DeleteTablet for 3 replicas of tablet e0da5de3bac54a4abf463be8650e9c29
I20260926 20:18:30.662101 13665 tablet_service.cc:1514] Processing DeleteTablet for tablet e0da5de3bac54a4abf463be8650e9c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:30 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:36444
I20260926 20:18:30.662145 13933 tablet_service.cc:1514] Processing DeleteTablet for tablet e0da5de3bac54a4abf463be8650e9c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:30 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:55756
I20260926 20:18:30.662201 13799 tablet_service.cc:1514] Processing DeleteTablet for tablet e0da5de3bac54a4abf463be8650e9c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:30 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:45686
I20260926 20:18:30.662703 14041 tablet_replica.cc:330] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: stopping tablet replica
I20260926 20:18:30.662940 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 13602
I20260926 20:18:30.663072 14041 raft_consensus.cc:2229] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:18:30.664157 14041 raft_consensus.cc:2258] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:18:30.666286 14041 ts_tablet_manager.cc:1905] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:18:30.668493 14041 ts_tablet_manager.cc:1918] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:18:30.668612 14041 log.cc:1196] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/e0da5de3bac54a4abf463be8650e9c29
I20260926 20:18:30.668918 14041 ts_tablet_manager.cc:1939] T e0da5de3bac54a4abf463be8650e9c29 P d08241111ed74731884a8c87ebcc0ed5: Deleting consensus metadata
I20260926 20:18:30.670287 14039 tablet_replica.cc:330] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: stopping tablet replica
I20260926 20:18:30.670271 13524 catalog_manager.cc:4841] TS d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057): tablet e0da5de3bac54a4abf463be8650e9c29 (table test-table [id=083c2d13a2fb4c31af424b6f2b12c024]) successfully deleted
I20260926 20:18:30.670662 14039 raft_consensus.cc:2229] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:18:30.671020 14039 raft_consensus.cc:2258] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:18:30.672281 14039 ts_tablet_manager.cc:1905] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:18:30.674437 13524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:44169 (error 108)
I20260926 20:18:30.674642 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 13736
I20260926 20:18:30.676319 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:30.676013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:44169 (local address 127.0.0.1:36456)
0926 20:18:30.676148 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:30.676224 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:18:30.676493 13524 catalog_manager.cc:4584] TS a5f85a54b3d5432f8c2089249cc619a7 (127.31.119.193:44169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0da5de3bac54a4abf463be8650e9c29: Network error: Client connection negotiation failed: client connection to 127.31.119.193:44169: connect: Connection refused (error 111)
I20260926 20:18:30.677371 14039 ts_tablet_manager.cc:1918] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:18:30.677508 14039 log.cc:1196] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/e0da5de3bac54a4abf463be8650e9c29
I20260926 20:18:30.677829 14039 ts_tablet_manager.cc:1939] T e0da5de3bac54a4abf463be8650e9c29 P 43e93296fd254148a3023e132d7ee77a: Deleting consensus metadata
I20260926 20:18:30.678915 13527 catalog_manager.cc:4841] TS 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403): tablet e0da5de3bac54a4abf463be8650e9c29 (table test-table [id=083c2d13a2fb4c31af424b6f2b12c024]) successfully deleted
I20260926 20:18:30.681602 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 13870
I20260926 20:18:30.687069 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 13508
I20260926 20:18:30.693658 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38497
--webserver_interface=127.31.119.254
--webserver_port=33991
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:38497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:30.815474 14043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:30.815764 14043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:30.815807 14043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:30.819950 14043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:18:30.820016 14043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:30.820036 14043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:30.820053 14043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:18:30.820107 14043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:18:30.825528 14043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:38497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=33991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:30.826658 14043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:30.828011 14043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:30.842119 14049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:30.842146 14052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:30.842127 14050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:30.842485 14043 server_base.cc:1034] running on GCE node
I20260926 20:18:30.843309 14043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:30.844674 14043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:30.845887 14043 hybrid_clock.cc:648] HybridClock initialized: now 1790453910845846 us; error 57 us; skew 500 ppm
Sep 26 20:18:30 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453910, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:30.848996 14043 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:18:30.850075 14043 webserver.cc:460] Webserver started at http://127.31.119.254:33991/ using document root <none> and password file <none>
I20260926 20:18:30.850682 14043 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:30.850802 14043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:30.854945 14043 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:18:30.857273 14059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:30.858901 14043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:30.859076 14043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "d00f1204703746c69cd6f343416750f2"
format_stamp: "Formatted at 2026-09-26 20:18:29 on dist-test-slave-vnbh"
server_key: "4be6caa1c1c995e566f328af06c51aca"
server_key_iv: "00af73ef91528402b84789f1ed3d6cbc"
server_key_version: "encryptionkey@0"
I20260926 20:18:30.859768 14043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:30.875126 14043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:30.879062 14043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:30.879302 14043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:30.887758 14043 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:38497
I20260926 20:18:30.888151 14111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:38497 every 8 connection(s)
I20260926 20:18:30.889271 14043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:18:30.890393 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 14043
I20260926 20:18:30.890889 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:44169
--local_ip_for_outbound_sockets=127.31.119.193
--tserver_master_addrs=127.31.119.254:38497
--webserver_port=43875
--webserver_interface=127.31.119.193
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:18:30.895279 14112 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:18:30.898047 14112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Bootstrap starting.
I20260926 20:18:30.921751 14112 log.cc:824] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:30.932821 14112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:18:30.933275 14112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Bootstrap complete.
I20260926 20:18:30.939088 14112 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:30.939628 14112 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00f1204703746c69cd6f343416750f2, State: Initialized, Role: FOLLOWER
I20260926 20:18:30.940208 14112 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:30.940357 14112 raft_consensus.cc:388] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:18:30.940423 14112 raft_consensus.cc:482] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:18:30.940512 14112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:18:30.942943 14112 raft_consensus.cc:504] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:30.943325 14112 leader_election.cc:302] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d00f1204703746c69cd6f343416750f2; no voters:
I20260926 20:18:30.943655 14112 leader_election.cc:288] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:18:30.943799 14116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:18:30.944057 14116 raft_consensus.cc:686] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 2 LEADER]: Becoming Leader. State: Replica: d00f1204703746c69cd6f343416750f2, State: Running, Role: LEADER
I20260926 20:18:30.944386 14116 consensus_queue.cc:237] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:30.944979 14112 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:18:30.946067 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d00f1204703746c69cd6f343416750f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } } }
I20260926 20:18:30.946096 14118 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00f1204703746c69cd6f343416750f2. Latest consensus state: current_term: 2 leader_uuid: "d00f1204703746c69cd6f343416750f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } } }
I20260926 20:18:30.946223 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:30.946223 14118 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:30.950652 14131 catalog_manager.cc:1259] Loaded cluster ID: 1c37ec29392245fca3efc25bf77ce73a
I20260926 20:18:30.950726 14131 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: loading cluster ID for follower catalog manager: success
W20260926 20:18:30.951694 14131 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260926 20:18:30.951779 14131 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260926 20:18:30.951862 14125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:18:30.954669 14125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:18:30.955152 14125 catalog_manager.cc:1259] Loaded cluster ID: 1c37ec29392245fca3efc25bf77ce73a
I20260926 20:18:30.955211 14125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:18:30.956203 14125 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 20:18:30.960645 14125 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 20:18:30.966691 14125 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:18:30.967535 14125 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Loaded TSK: 0
I20260926 20:18:30.968832 14125 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:18:31.028699 14114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:31.029027 14114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:31.029129 14114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:31.033479 14114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:31.033571 14114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:31.033674 14114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:18:31.039669 14114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:44169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=43875
--tserver_master_addrs=127.31.119.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:31.040849 14114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:31.042008 14114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:31.053828 14142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:31.053884 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:31.053859 14139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:31.054509 14114 server_base.cc:1034] running on GCE node
I20260926 20:18:31.055006 14114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:31.055774 14114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:31.056993 14114 hybrid_clock.cc:648] HybridClock initialized: now 1790453911056944 us; error 66 us; skew 500 ppm
Sep 26 20:18:31 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:31.060225 14114 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:18:31.061352 14114 webserver.cc:460] Webserver started at http://127.31.119.193:43875/ using document root <none> and password file <none>
I20260926 20:18:31.061990 14114 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:31.062065 14114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:31.066473 14114 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:18:31.068835 14149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:31.070255 14114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:31.070359 14114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "a5f85a54b3d5432f8c2089249cc619a7"
format_stamp: "Formatted at 2026-09-26 20:18:29 on dist-test-slave-vnbh"
server_key: "5c8db35f4bcbb3f1bc556710d0eefc88"
server_key_iv: "c24067995d0fd81671c5973d6b352083"
server_key_version: "encryptionkey@0"
I20260926 20:18:31.070973 14114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:31.098354 14114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:31.102494 14114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:31.102782 14114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:31.103511 14114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:31.104848 14156 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:18:31.109405 14114 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:18:31.109479 14114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 20:18:31.109565 14114 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:18:31.111402 14114 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:18:31.111469 14114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 20:18:31.111887 14156 tablet_bootstrap.cc:492] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Bootstrap starting.
I20260926 20:18:31.123199 14114 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:44169
I20260926 20:18:31.123214 14263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:44169 every 8 connection(s)
I20260926 20:18:31.124526 14114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:18:31.129405 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 14114
I20260926 20:18:31.129865 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:33057
--local_ip_for_outbound_sockets=127.31.119.194
--tserver_master_addrs=127.31.119.254:38497
--webserver_port=37761
--webserver_interface=127.31.119.194
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:18:31.134414 14156 log.cc:824] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:18:31 dist-test-slave-vnbh krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:31.144054 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.126850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:41163 (local address 127.31.119.254:38497)
0926 20:18:31.127242 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.127258 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.128475 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 20:18:31.130372 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.130390 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.130813 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.131179 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:31.132815 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.134038 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.135571 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.136059 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.136246 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.138922 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:31.138927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:31.138942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:31.138972 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:31.140786 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:31.141367 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:31.141381 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:31.141385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:31.141517 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:31.141874 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:31.141882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:31.141883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:31.142229 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:31.142375 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.142903 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.143158 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":155,"threads_started":1}
I20260926 20:18:31.144068 14267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.126934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:41163)
0926 20:18:31.128257 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:31.128315 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:31.129636 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:31.131640 (+ 2004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:31.131647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:31.131997 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:31.132579 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.132604 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.134210 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.134213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.135413 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.135428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.136214 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.136218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.136404 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.137049 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:31.137066 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:31.138714 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:31.140926 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:31.140932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:31.140953 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:31.141216 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:31.141621 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:31.141626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:31.141628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:31.141766 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:31.142347 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:31.142353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:31.142599 (+ 246us) client_negotiation.cc:769] Sending connection context
0926 20:18:31.142776 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 20:18:31.143137 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"thread_start_us":156,"threads_started":1}
I20260926 20:18:31.144586 14156 tablet_bootstrap.cc:492] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:18:31.145210 14156 tablet_bootstrap.cc:492] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Bootstrap complete.
I20260926 20:18:31.146664 14156 ts_tablet_manager.cc:1397] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Time spent bootstrapping tablet: real 0.035s user 0.029s sys 0.005s
I20260926 20:18:31.147686 14264 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38497
I20260926 20:18:31.147895 14264 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:31.148427 14264 heartbeater.cc:511] Master 127.31.119.254:38497 requested a full tablet report, sending...
I20260926 20:18:31.149658 14156 raft_consensus.cc:348] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:31.150218 14156 raft_consensus.cc:729] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f85a54b3d5432f8c2089249cc619a7, State: Initialized, Role: FOLLOWER
I20260926 20:18:31.150698 14156 consensus_queue.cc:260] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } } peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } }
I20260926 20:18:31.151156 14076 ts_manager.cc:194] Registered new tserver with Master: a5f85a54b3d5432f8c2089249cc619a7 (127.31.119.193:44169)
I20260926 20:18:31.151619 14156 ts_tablet_manager.cc:1428] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:18:31.155367 14076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:41163
I20260926 20:18:31.157945 14264 heartbeater.cc:503] Master 127.31.119.254:38497 was elected leader, sending a full tablet report...
Sep 26 20:18:31 dist-test-slave-vnbh krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453910, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:18:31.166263 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.154664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:44169 (local address 127.0.0.1:36470)
0926 20:18:31.155332 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:31.155368 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:31.155512 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:31.156162 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:31.156168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:31.156604 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:31.156733 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.156741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.158108 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.158112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.159047 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.159058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.159834 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.159837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.159918 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.160502 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:31.160514 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:31.161918 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:31.164243 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:31.164251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:31.164256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:31.164518 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:31.164963 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:31.164969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:31.164971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:31.165076 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:31.165684 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:31.165690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:31.165807 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 20:18:31.165948 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 20:18:31.166120 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":117,"threads_started":1}
I20260926 20:18:31.166411 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.154711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.31.119.193:44169)
0926 20:18:31.155768 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.155776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.155793 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:18:31.155852 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.155855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.156015 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.156162 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:31.156872 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.157968 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.159196 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.159717 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.159824 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.162068 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:31.162079 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:31.162093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:31.162120 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:31.164110 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:31.164709 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:31.164716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:31.164718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:31.164829 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:31.165180 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:31.165183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:31.165185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:31.165558 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:31.165690 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.166053 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.166248 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975,"thread_start_us":103,"threads_started":1}
I20260926 20:18:31.167800 14198 tablet_service.cc:1514] Processing DeleteTablet for tablet e0da5de3bac54a4abf463be8650e9c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:30 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:36470
I20260926 20:18:31.168857 14273 tablet_replica.cc:330] stopping tablet replica
I20260926 20:18:31.169222 14273 raft_consensus.cc:2229] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:18:31.169548 14273 raft_consensus.cc:2258] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:18:31.170620 14273 ts_tablet_manager.cc:1905] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:18:31.174619 14273 ts_tablet_manager.cc:1918] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:18:31.174749 14273 log.cc:1196] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/e0da5de3bac54a4abf463be8650e9c29
I20260926 20:18:31.175058 14273 ts_tablet_manager.cc:1939] T e0da5de3bac54a4abf463be8650e9c29 P a5f85a54b3d5432f8c2089249cc619a7: Deleting consensus metadata
I20260926 20:18:31.177459 14060 catalog_manager.cc:4841] TS a5f85a54b3d5432f8c2089249cc619a7 (127.31.119.193:44169): tablet e0da5de3bac54a4abf463be8650e9c29 (table test-table [id=083c2d13a2fb4c31af424b6f2b12c024]) successfully deleted
W20260926 20:18:31.261762 14268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:31.262121 14268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:31.262216 14268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:31.266584 14268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:31.266688 14268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:31.266793 14268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:18:31.272321 14268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:33057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=37761
--tserver_master_addrs=127.31.119.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:31.273516 14268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:31.274696 14268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:31.285454 14281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:31.285430 14279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:31.285408 14278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:31.285496 14268 server_base.cc:1034] running on GCE node
I20260926 20:18:31.286115 14268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:31.286839 14268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:31.288024 14268 hybrid_clock.cc:648] HybridClock initialized: now 1790453911288009 us; error 26 us; skew 500 ppm
Sep 26 20:18:31 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:31.291225 14268 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:18:31.292243 14268 webserver.cc:460] Webserver started at http://127.31.119.194:37761/ using document root <none> and password file <none>
I20260926 20:18:31.292830 14268 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:31.292933 14268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:31.297248 14268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:31.299909 14288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:31.301564 14268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:31.301729 14268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "d08241111ed74731884a8c87ebcc0ed5"
format_stamp: "Formatted at 2026-09-26 20:18:30 on dist-test-slave-vnbh"
server_key: "e3720e159899c9520723260cdb4dedad"
server_key_iv: "928e1c10241662d70a8d1e081f393f12"
server_key_version: "encryptionkey@0"
I20260926 20:18:31.302368 14268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:31.317584 14268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:31.320887 14268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:31.321113 14268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:31.321890 14268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:31.322810 14268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:31.322897 14268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:31.322978 14268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:31.323042 14268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:31.333206 14268 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:33057
I20260926 20:18:31.333225 14401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:33057 every 8 connection(s)
I20260926 20:18:31.334443 14268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:18:31.338568 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 14268
I20260926 20:18:31.339092 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:46403
--local_ip_for_outbound_sockets=127.31.119.195
--tserver_master_addrs=127.31.119.254:38497
--webserver_port=37551
--webserver_interface=127.31.119.195
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:18:31 dist-test-slave-vnbh krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:31.351425 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.336383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:40251 (local address 127.31.119.254:38497)
0926 20:18:31.336537 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.336540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.337621 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 20:18:31.340018 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.340025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.340107 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.340217 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:31.342094 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.342690 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.344089 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.344331 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.344453 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.347460 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:31.347464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:31.347466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:31.347494 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:31.349140 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:31.349750 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:31.349753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:31.349757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:31.349842 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:31.350220 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:31.350223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:31.350225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:31.350430 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:31.350538 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.351098 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.351248 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:18:31.352317 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.336810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:40251)
0926 20:18:31.337436 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:31.337503 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:31.339725 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:31.340449 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:31.340458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:31.341051 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:31.341865 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.341889 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.342848 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.342851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.343920 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.343929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.344534 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.344539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.344754 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.345452 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:31.345471 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:31.347260 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:31.349306 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:31.349313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:31.349333 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:31.349629 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:31.349940 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:31.349945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:31.349949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:31.350112 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:31.350546 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:31.350556 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:31.350839 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 20:18:31.351041 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 20:18:31.351409 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":117,"threads_started":1}
I20260926 20:18:31.353757 14402 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38497
I20260926 20:18:31.354053 14402 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:31.354606 14402 heartbeater.cc:511] Master 127.31.119.254:38497 requested a full tablet report, sending...
I20260926 20:18:31.355834 14076 ts_manager.cc:194] Registered new tserver with Master: d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057)
I20260926 20:18:31.356693 14076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:40251
W20260926 20:18:31.462659 14405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:31.462960 14405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:31.463006 14405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:31.467350 14405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:31.467419 14405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:31.467482 14405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:18:31.472970 14405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:46403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=37551
--tserver_master_addrs=127.31.119.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:31.474129 14405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:31.475190 14405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:18:31.485673 14405 server_base.cc:1034] running on GCE node
W20260926 20:18:31.485634 14410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:31.485634 14411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:31.485661 14413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:31.486543 14405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:31.487255 14405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:31.488471 14405 hybrid_clock.cc:648] HybridClock initialized: now 1790453911488450 us; error 35 us; skew 500 ppm
Sep 26 20:18:31 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:31.491495 14405 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:18:31.492443 14405 webserver.cc:460] Webserver started at http://127.31.119.195:37551/ using document root <none> and password file <none>
I20260926 20:18:31.493029 14405 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:31.493142 14405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:31.497118 14405 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:18:31.499469 14420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:31.500977 14405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:18:31.501145 14405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "43e93296fd254148a3023e132d7ee77a"
format_stamp: "Formatted at 2026-09-26 20:18:30 on dist-test-slave-vnbh"
server_key: "9fc8bfd6f0127c3829872e9fab8be4cd"
server_key_iv: "538a1f488d31eb365bcfc21d285a70a9"
server_key_version: "encryptionkey@0"
I20260926 20:18:31.501813 14405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:31.516665 14405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:31.520205 14405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:31.520460 14405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:31.521128 14405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:31.522079 14405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:31.522153 14405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:31.522231 14405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:31.522274 14405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:31.531955 14405 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:46403
I20260926 20:18:31.531968 14533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:46403 every 8 connection(s)
I20260926 20:18:31.533071 14405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:18:31.537997 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 14405
Sep 26 20:18:31 dist-test-slave-vnbh krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:31.548187 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.538745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.31.119.254:38497)
0926 20:18:31.539001 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.539005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.539355 (+ 350us) server_negotiation.cc:407] Connection header received
0926 20:18:31.539500 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.539503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.539545 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.539631 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:31.540366 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.540891 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.542072 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.542379 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.542466 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.544306 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:31.544313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:31.544315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:31.544347 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:31.546283 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:31.546729 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:31.546735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:31.546740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:31.546858 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:31.547149 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:31.547152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:31.547154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:31.547430 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:31.547638 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.547843 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.547973 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
I20260926 20:18:31.549870 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.535196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:38347 (local address 127.31.119.254:38497)
0926 20:18:31.535393 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.535397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.536351 (+ 954us) server_negotiation.cc:407] Connection header received
0926 20:18:31.537919 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.537922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.537985 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.538092 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:31.539724 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.540333 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.541859 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.542184 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.542286 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.545772 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:31.545777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:31.545780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:31.545824 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:31.547503 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:31.548040 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:31.548043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:31.548045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:31.548150 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:31.548551 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:31.548554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:31.548556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:31.548781 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:31.548894 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.549595 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.549712 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 20:18:31.550833 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.535586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:38347)
0926 20:18:31.536165 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:31.536217 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:31.537684 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:31.538286 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:31.538293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:31.538753 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:31.539548 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.539566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.540456 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.540459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.541685 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.541696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.542340 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.542344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.542539 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.543288 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:31.543317 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:31.545535 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:31.547625 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:31.547632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:31.547651 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:31.547926 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:31.548233 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:31.548236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:31.548238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:31.548426 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:31.548906 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:31.548913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:31.549272 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 20:18:31.549508 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 20:18:31.549784 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":198,"threads_started":1}
I20260926 20:18:31.552170 14534 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38497
I20260926 20:18:31.552416 14534 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:31.552910 14534 heartbeater.cc:511] Master 127.31.119.254:38497 requested a full tablet report, sending...
I20260926 20:18:31.554009 14076 ts_manager.cc:194] Registered new tserver with Master: 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403)
I20260926 20:18:31.554548 14076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:38347
I20260926 20:18:31.562654 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:18:31.567409 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.563313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.31.119.254:38497)
0926 20:18:31.563455 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.563458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.563678 (+ 220us) server_negotiation.cc:407] Connection header received
0926 20:18:31.563886 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.563888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.564041 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.564134 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:18:31.564341 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.564937 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.565891 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.566135 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.566218 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.566545 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:18:31.567049 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:18:31.567146 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.567238 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.567292 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:18:31.568166 14076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:18:31.572010 14076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:18:31.580770 14198 tablet_service.cc:1467] Processing CreateTablet for tablet 5e55f6773bd140faa9a6b60900643be9 (DEFAULT_TABLE table=test-table [id=146cf2555aaa437995b572d2cf4add6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:31.581519 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e55f6773bd140faa9a6b60900643be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:31.584564 14156 tablet_bootstrap.cc:492] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Bootstrap starting.
I20260926 20:18:31.584790 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.579175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:33057 (local address 127.0.0.1:45698)
0926 20:18:31.579360 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:31.579373 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:31.579481 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:31.580616 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:31.580619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:31.580639 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:31.580751 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.580757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.582219 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.582222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.583432 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.583441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.584314 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.584317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.584409 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.584456 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:18:31.584542 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:18:31.584631 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":6}
I20260926 20:18:31.584836 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.579380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.31.119.194:33057)
0926 20:18:31.580057 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.580070 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.580095 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:18:31.580183 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.580189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.580423 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.580605 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:31.580897 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.582070 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.583560 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.584179 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.584325 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.584468 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.584574 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.584670 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":150,"threads_started":1}
I20260926 20:18:31.585050 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.579444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:46403 (local address 127.0.0.1:55766)
0926 20:18:31.579696 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:31.579709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:31.579799 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:31.580801 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:31.580806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:31.580829 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:31.580903 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.580907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.582303 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.582306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.583471 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.583485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.584643 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.584647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.584754 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.584780 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:18:31.584832 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:18:31.584904 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
I20260926 20:18:31.585559 14156 tablet_bootstrap.cc:654] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:31.586233 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.579654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.31.119.195:46403)
0926 20:18:31.580117 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.580128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.580146 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:18:31.580206 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.580210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.580423 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.580606 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:31.581158 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.582185 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.583646 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.584499 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.585708 (+ 1209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.585870 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.585937 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.586071 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":143,"threads_started":1}
I20260926 20:18:31.587136 14156 tablet_bootstrap.cc:492] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: No bootstrap required, opened a new log
I20260926 20:18:31.587217 14156 ts_tablet_manager.cc:1397] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:18:31.587668 14156 raft_consensus.cc:348] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:31.587786 14156 raft_consensus.cc:374] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:31.587821 14156 raft_consensus.cc:729] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f85a54b3d5432f8c2089249cc619a7, State: Initialized, Role: FOLLOWER
I20260926 20:18:31.587998 14156 consensus_queue.cc:260] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:31.588274 14156 ts_tablet_manager.cc:1428] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:18:31.588382 14336 tablet_service.cc:1467] Processing CreateTablet for tablet 5e55f6773bd140faa9a6b60900643be9 (DEFAULT_TABLE table=test-table [id=146cf2555aaa437995b572d2cf4add6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:31.589566 14336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e55f6773bd140faa9a6b60900643be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:31.589250 14468 tablet_service.cc:1467] Processing CreateTablet for tablet 5e55f6773bd140faa9a6b60900643be9 (DEFAULT_TABLE table=test-table [id=146cf2555aaa437995b572d2cf4add6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:31.590346 14468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e55f6773bd140faa9a6b60900643be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:31.596788 14547 tablet_bootstrap.cc:492] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: Bootstrap starting.
I20260926 20:18:31.597105 14548 tablet_bootstrap.cc:492] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: Bootstrap starting.
I20260926 20:18:31.599431 14547 tablet_bootstrap.cc:654] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:31.599431 14548 tablet_bootstrap.cc:654] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:31.600270 14547 log.cc:824] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:31.600270 14548 log.cc:824] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:31.602361 14547 tablet_bootstrap.cc:492] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: No bootstrap required, opened a new log
I20260926 20:18:31.602375 14548 tablet_bootstrap.cc:492] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: No bootstrap required, opened a new log
I20260926 20:18:31.602594 14547 ts_tablet_manager.cc:1397] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:18:31.602604 14548 ts_tablet_manager.cc:1397] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:18:31.605829 14548 raft_consensus.cc:348] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:31.605930 14547 raft_consensus.cc:348] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:31.606155 14548 raft_consensus.cc:374] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:31.606210 14548 raft_consensus.cc:729] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e93296fd254148a3023e132d7ee77a, State: Initialized, Role: FOLLOWER
I20260926 20:18:31.606213 14547 raft_consensus.cc:374] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:31.606271 14547 raft_consensus.cc:729] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08241111ed74731884a8c87ebcc0ed5, State: Initialized, Role: FOLLOWER
I20260926 20:18:31.606732 14548 consensus_queue.cc:260] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:31.607254 14547 consensus_queue.cc:260] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:31.607482 14534 heartbeater.cc:503] Master 127.31.119.254:38497 was elected leader, sending a full tablet report...
I20260926 20:18:31.607970 14402 heartbeater.cc:503] Master 127.31.119.254:38497 was elected leader, sending a full tablet report...
I20260926 20:18:31.607969 14548 ts_tablet_manager.cc:1428] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:18:31.608294 14547 ts_tablet_manager.cc:1428] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 20:18:31.626046 14265 tablet.cc:2369] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:18:31.740343 14551 raft_consensus.cc:482] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:18:31.740588 14551 raft_consensus.cc:504] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:31.741940 14551 leader_election.cc:288] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057), a5f85a54b3d5432f8c2089249cc619a7 (127.31.119.193:44169)
I20260926 20:18:31.746563 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.742275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:33057 (local address 127.31.119.195:39025)
0926 20:18:31.742406 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:31.742420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:31.742556 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:31.742829 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:31.742833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:31.742858 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:31.743014 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.743022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.743977 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.743981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.745080 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.745090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.746084 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.746088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.746201 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.746227 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:18:31.746312 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:18:31.746387 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 20:18:31.746609 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.742405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:39025 (local address 127.31.119.194:33057)
0926 20:18:31.742561 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.742565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.742584 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:18:31.742649 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.742652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.742708 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.742789 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:31.743154 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.743864 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.745225 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.745973 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.746073 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.746117 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.746369 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.746486 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:18:31.747215 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.742638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:44169 (local address 127.31.119.195:35191)
0926 20:18:31.743102 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:31.743118 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:31.743208 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:31.743496 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:31.743499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:31.743517 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:31.743630 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.743636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.744606 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.744608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.745682 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:31.745692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.746808 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.746815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:31.746916 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.746956 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:18:31.747017 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:18:31.747090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":144,"threads_started":1}
I20260926 20:18:31.747447 14356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e55f6773bd140faa9a6b60900643be9" candidate_uuid: "43e93296fd254148a3023e132d7ee77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08241111ed74731884a8c87ebcc0ed5" is_pre_election: true
I20260926 20:18:31.747790 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.742852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:35191 (local address 127.31.119.193:44169)
0926 20:18:31.743155 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.743159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.743173 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:18:31.743316 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.743318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.743373 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.743467 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:31.743732 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.744472 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.745835 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.746586 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.747345 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.747533 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.747566 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.747653 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
I20260926 20:18:31.747802 14356 raft_consensus.cc:2454] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e93296fd254148a3023e132d7ee77a in term 0.
I20260926 20:18:31.748255 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e55f6773bd140faa9a6b60900643be9" candidate_uuid: "43e93296fd254148a3023e132d7ee77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f85a54b3d5432f8c2089249cc619a7" is_pre_election: true
I20260926 20:18:31.748308 14421 leader_election.cc:302] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e93296fd254148a3023e132d7ee77a, d08241111ed74731884a8c87ebcc0ed5; no voters:
I20260926 20:18:31.748540 14218 raft_consensus.cc:2454] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e93296fd254148a3023e132d7ee77a in term 0.
I20260926 20:18:31.748610 14551 raft_consensus.cc:2781] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:18:31.748723 14551 raft_consensus.cc:482] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:18:31.748797 14551 raft_consensus.cc:3037] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:31.750066 14551 raft_consensus.cc:504] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:31.750416 14551 leader_election.cc:288] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [CANDIDATE]: Term 1 election: Requested vote from peers d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057), a5f85a54b3d5432f8c2089249cc619a7 (127.31.119.193:44169)
I20260926 20:18:31.750810 14356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e55f6773bd140faa9a6b60900643be9" candidate_uuid: "43e93296fd254148a3023e132d7ee77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08241111ed74731884a8c87ebcc0ed5"
I20260926 20:18:31.750844 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e55f6773bd140faa9a6b60900643be9" candidate_uuid: "43e93296fd254148a3023e132d7ee77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f85a54b3d5432f8c2089249cc619a7"
I20260926 20:18:31.750919 14356 raft_consensus.cc:3037] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:31.750988 14218 raft_consensus.cc:3037] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:31.751960 14218 raft_consensus.cc:2454] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e93296fd254148a3023e132d7ee77a in term 1.
I20260926 20:18:31.752100 14356 raft_consensus.cc:2454] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e93296fd254148a3023e132d7ee77a in term 1.
I20260926 20:18:31.752292 14421 leader_election.cc:302] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e93296fd254148a3023e132d7ee77a, a5f85a54b3d5432f8c2089249cc619a7; no voters:
I20260926 20:18:31.752498 14551 raft_consensus.cc:2781] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:18:31.752736 14551 raft_consensus.cc:686] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 1 LEADER]: Becoming Leader. State: Replica: 43e93296fd254148a3023e132d7ee77a, State: Running, Role: LEADER
I20260926 20:18:31.753120 14551 consensus_queue.cc:237] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:31.756410 14076 catalog_manager.cc:5495] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e93296fd254148a3023e132d7ee77a (127.31.119.195). New cstate: current_term: 1 leader_uuid: "43e93296fd254148a3023e132d7ee77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } health_report { overall_health: UNKNOWN } } }
I20260926 20:18:31.782864 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.779089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.31.119.195:46403)
0926 20:18:31.779236 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:31.779242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:31.779260 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:18:31.779358 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:31.779360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:31.779416 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:31.779536 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:18:31.779778 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.780298 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.781326 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:31.781542 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:31.781652 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:31.781990 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:18:31.782443 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:18:31.782598 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:31.782643 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 20:18:31.782704 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 20:18:31.784080 14535 tablet.cc:2369] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:18:31.789510 14218 raft_consensus.cc:1261] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Refusing update from remote peer 43e93296fd254148a3023e132d7ee77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:18:31.789510 14356 raft_consensus.cc:1261] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5 [term 1 FOLLOWER]: Refusing update from remote peer 43e93296fd254148a3023e132d7ee77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:18:31.790336 14555 consensus_queue.cc:1035] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:18:31.790598 14551 consensus_queue.cc:1035] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:18:31.797849 14557 mvcc.cc:205] Tried to move back new op lower bound from 7333699222681071616 to 7333699222543527936. Current Snapshot: MvccSnapshot[applied={T|T < 7333699222681071616}]
I20260926 20:18:31.798914 14559 mvcc.cc:205] Tried to move back new op lower bound from 7333699222681071616 to 7333699222543527936. Current Snapshot: MvccSnapshot[applied={T|T < 7333699222681071616}]
I20260926 20:18:31.807622 14075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:18:31.807811 14075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:18:31.810385 14075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Sending DeleteTablet for 3 replicas of tablet 5e55f6773bd140faa9a6b60900643be9
I20260926 20:18:31.811183 14198 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e55f6773bd140faa9a6b60900643be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:31 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:36470
I20260926 20:18:31.811247 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 14114
I20260926 20:18:31.811246 14468 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e55f6773bd140faa9a6b60900643be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:31 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:55766
I20260926 20:18:31.812094 14336 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e55f6773bd140faa9a6b60900643be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:31 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:45698
I20260926 20:18:31.812597 14567 tablet_replica.cc:330] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: stopping tablet replica
I20260926 20:18:31.812714 14566 tablet_replica.cc:330] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: stopping tablet replica
I20260926 20:18:31.813052 14566 raft_consensus.cc:2229] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:18:31.813052 14567 raft_consensus.cc:2229] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:18:31.813493 14566 raft_consensus.cc:2258] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:18:31.813493 14567 raft_consensus.cc:2258] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:18:31.815128 14566 ts_tablet_manager.cc:1905] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:18:31.815128 14567 ts_tablet_manager.cc:1905] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:18:31.818032 14566 ts_tablet_manager.cc:1918] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:18:31.818032 14567 ts_tablet_manager.cc:1918] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:18:31.818150 14566 log.cc:1196] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/5e55f6773bd140faa9a6b60900643be9
I20260926 20:18:31.818382 14567 log.cc:1196] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/5e55f6773bd140faa9a6b60900643be9
W20260926 20:18:31.818588 14060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:44169 (error 108)
I20260926 20:18:31.818898 14567 ts_tablet_manager.cc:1939] T 5e55f6773bd140faa9a6b60900643be9 P d08241111ed74731884a8c87ebcc0ed5: Deleting consensus metadata
I20260926 20:18:31.819223 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 14268
I20260926 20:18:31.819247 14566 ts_tablet_manager.cc:1939] T 5e55f6773bd140faa9a6b60900643be9 P 43e93296fd254148a3023e132d7ee77a: Deleting consensus metadata
I20260926 20:18:31.820966 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.820636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:44169 (local address 127.0.0.1:36484)
0926 20:18:31.820779 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:31.820869 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:44169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 20:18:31.821116 14060 catalog_manager.cc:4584] TS a5f85a54b3d5432f8c2089249cc619a7 (127.31.119.193:44169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e55f6773bd140faa9a6b60900643be9: Network error: Client connection negotiation failed: client connection to 127.31.119.193:44169: connect: Connection refused (error 111)
I20260926 20:18:31.822379 14063 catalog_manager.cc:4841] TS 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403): tablet 5e55f6773bd140faa9a6b60900643be9 (table test-table [id=146cf2555aaa437995b572d2cf4add6c]) successfully deleted
I20260926 20:18:31.826566 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 14405
I20260926 20:18:31.828637 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:31.826967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:33057 (local address 127.0.0.1:45710)
0926 20:18:31.828491 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:31.828551 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.194:33057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1399}
I20260926 20:18:31.832187 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 14043
I20260926 20:18:31.838160 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38497
--webserver_interface=127.31.119.254
--webserver_port=33991
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:38497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:31.968739 14569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:31.969027 14569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:31.969070 14569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:31.973104 14569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:18:31.973166 14569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:31.973186 14569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:31.973203 14569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:18:31.973246 14569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:18:31.978691 14569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:38497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=33991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:31.979969 14569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:31.981077 14569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:31.990253 14577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:31.990188 14569 server_base.cc:1034] running on GCE node
W20260926 20:18:31.990201 14574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:31.990226 14575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:31.991123 14569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:31.992345 14569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:31.993542 14569 hybrid_clock.cc:648] HybridClock initialized: now 1790453911993525 us; error 36 us; skew 500 ppm
Sep 26 20:18:31 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453911, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:31.996469 14569 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:18:31.997468 14569 webserver.cc:460] Webserver started at http://127.31.119.254:33991/ using document root <none> and password file <none>
I20260926 20:18:31.998052 14569 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:31.998116 14569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:32.002316 14569 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:18:32.004961 14584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:32.006487 14569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:18:32.006608 14569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "d00f1204703746c69cd6f343416750f2"
format_stamp: "Formatted at 2026-09-26 20:18:29 on dist-test-slave-vnbh"
server_key: "4be6caa1c1c995e566f328af06c51aca"
server_key_iv: "00af73ef91528402b84789f1ed3d6cbc"
server_key_version: "encryptionkey@0"
I20260926 20:18:32.007192 14569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:32.037464 14569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:32.041000 14569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:32.041162 14569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:32.048753 14569 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:38497
I20260926 20:18:32.049098 14636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:38497 every 8 connection(s)
I20260926 20:18:32.050045 14569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:18:32.054697 14637 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:18:32.055930 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 14569
I20260926 20:18:32.056491 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:44169
--local_ip_for_outbound_sockets=127.31.119.193
--tserver_master_addrs=127.31.119.254:38497
--webserver_port=43875
--webserver_interface=127.31.119.193
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:18:32.056914 14637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Bootstrap starting.
I20260926 20:18:32.075140 14637 log.cc:824] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:32.086609 14637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:18:32.087057 14637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2: Bootstrap complete.
I20260926 20:18:32.091387 14637 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:32.091809 14637 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00f1204703746c69cd6f343416750f2, State: Initialized, Role: FOLLOWER
I20260926 20:18:32.092357 14637 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:32.092482 14637 raft_consensus.cc:388] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:18:32.092566 14637 raft_consensus.cc:482] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:18:32.092685 14637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:18:32.094713 14637 raft_consensus.cc:504] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:32.095068 14637 leader_election.cc:302] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d00f1204703746c69cd6f343416750f2; no voters:
I20260926 20:18:32.095396 14637 leader_election.cc:288] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:18:32.095556 14641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:18:32.095825 14641 raft_consensus.cc:686] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [term 3 LEADER]: Becoming Leader. State: Replica: d00f1204703746c69cd6f343416750f2, State: Running, Role: LEADER
I20260926 20:18:32.096169 14641 consensus_queue.cc:237] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } }
I20260926 20:18:32.096694 14637 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:18:32.097878 14643 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00f1204703746c69cd6f343416750f2. Latest consensus state: current_term: 3 leader_uuid: "d00f1204703746c69cd6f343416750f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } } }
I20260926 20:18:32.097888 14642 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d00f1204703746c69cd6f343416750f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00f1204703746c69cd6f343416750f2" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38497 } } }
I20260926 20:18:32.098008 14643 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:32.098013 14642 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00f1204703746c69cd6f343416750f2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:32.098380 14650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:18:32.104029 14650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:18:32.105000 14650 catalog_manager.cc:1259] Loaded cluster ID: 1c37ec29392245fca3efc25bf77ce73a
I20260926 20:18:32.105067 14650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 20:18:32.106139 14650 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6efb1d79ad google::LogMessage::Fail() at ??:0
@ 0x7f6efb1da88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6efb1d74de google::LogMessage::Flush() at ??:0
@ 0x7f6efb1d8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6f05b1afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6f05b1bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6f05b18579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6f05b5c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6efd4293d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6efd429c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6efd42b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6efd41a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6efea5e6db start_thread at ??:0
@ 0x7f6ef9cd561f clone at ??:0
W20260926 20:18:32.195729 14639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:32.196056 14639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:32.196154 14639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:32.200416 14639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:32.200511 14639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:32.200603 14639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:18:32.206425 14639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:44169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=43875
--tserver_master_addrs=127.31.119.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:32.207589 14639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:32.208684 14639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:32.219198 14681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:32.219220 14678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:32.219193 14679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:32.219717 14639 server_base.cc:1034] running on GCE node
I20260926 20:18:32.220217 14639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:32.220952 14639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:32.222154 14639 hybrid_clock.cc:648] HybridClock initialized: now 1790453912222105 us; error 67 us; skew 500 ppm
Sep 26 20:18:32 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:32.225260 14639 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:18:32.226414 14639 webserver.cc:460] Webserver started at http://127.31.119.193:43875/ using document root <none> and password file <none>
I20260926 20:18:32.227031 14639 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:32.227097 14639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:32.231277 14639 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:18:32.233779 14688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:32.235139 14639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:32.235268 14639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "a5f85a54b3d5432f8c2089249cc619a7"
format_stamp: "Formatted at 2026-09-26 20:18:29 on dist-test-slave-vnbh"
server_key: "5c8db35f4bcbb3f1bc556710d0eefc88"
server_key_iv: "c24067995d0fd81671c5973d6b352083"
server_key_version: "encryptionkey@0"
I20260926 20:18:32.235841 14639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:32.277982 14639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:32.281767 14639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:32.282107 14639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:32.282851 14639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:32.284186 14695 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:18:32.290242 14639 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:18:32.290352 14639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 20:18:32.290449 14639 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:18:32.292457 14639 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:18:32.292537 14639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 20:18:32.292901 14695 tablet_bootstrap.cc:492] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Bootstrap starting.
I20260926 20:18:32.303819 14639 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:44169
I20260926 20:18:32.303889 14802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:44169 every 8 connection(s)
I20260926 20:18:32.305086 14639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:18:32.314599 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 14639
I20260926 20:18:32.315155 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:33057
--local_ip_for_outbound_sockets=127.31.119.194
--tserver_master_addrs=127.31.119.254:38497
--webserver_port=37761
--webserver_interface=127.31.119.194
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:18:32.315917 14695 log.cc:824] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:32.323840 14695 tablet_bootstrap.cc:492] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:18:32.324254 14695 tablet_bootstrap.cc:492] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Bootstrap complete.
I20260926 20:18:32.325527 14695 ts_tablet_manager.cc:1397] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.018s
I20260926 20:18:32.328107 14695 raft_consensus.cc:348] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:32.328583 14695 raft_consensus.cc:729] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f85a54b3d5432f8c2089249cc619a7, State: Initialized, Role: FOLLOWER
I20260926 20:18:32.329103 14695 consensus_queue.cc:260] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:32.330125 14695 ts_tablet_manager.cc:1428] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260926 20:18:32.391894 14805 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:18:32.307273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:59665)
0926 20:18:32.307789 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:32.307859 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:32.308765 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:32.390987 (+ 82222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":151,"threads_started":1}
W20260926 20:18:32.392524 14803 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:38497 (0 consecutive failures): Network error: Failed to ping master at 127.31.119.254:38497: Client connection negotiation failed: client connection to 127.31.119.254:38497: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:18:32.393365 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:32.393019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:58213)
0926 20:18:32.393150 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:32.393221 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:18:32.394097 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:32.393890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:43623)
0926 20:18:32.393999 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:32.394051 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:18:32.394294 14803 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:18:32.448315 14806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:32.448625 14806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:32.448673 14806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:32.453152 14806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:32.453226 14806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:32.453346 14806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:18:32.459695 14806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:33057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=37761
--tserver_master_addrs=127.31.119.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:32.460932 14806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:32.462069 14806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:32.472754 14813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:32.472779 14816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:32.472776 14814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:32.473172 14806 server_base.cc:1034] running on GCE node
I20260926 20:18:32.473662 14806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:32.474411 14806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:32.475613 14806 hybrid_clock.cc:648] HybridClock initialized: now 1790453912475576 us; error 49 us; skew 500 ppm
Sep 26 20:18:32 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:32.478539 14806 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:18:32.479497 14806 webserver.cc:460] Webserver started at http://127.31.119.194:37761/ using document root <none> and password file <none>
I20260926 20:18:32.480103 14806 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:32.480170 14806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:32.484320 14806 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:18:32.486555 14823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:32.488005 14806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:18:32.488175 14806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "d08241111ed74731884a8c87ebcc0ed5"
format_stamp: "Formatted at 2026-09-26 20:18:30 on dist-test-slave-vnbh"
server_key: "e3720e159899c9520723260cdb4dedad"
server_key_iv: "928e1c10241662d70a8d1e081f393f12"
server_key_version: "encryptionkey@0"
I20260926 20:18:32.488794 14806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:32.509938 14806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:32.513439 14806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:32.513664 14806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:32.514340 14806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:32.515327 14806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:32.515406 14806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:32.515486 14806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:32.515543 14806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:32.526388 14806 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:33057
I20260926 20:18:32.526412 14936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:33057 every 8 connection(s)
I20260926 20:18:32.527606 14806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:18:32.531723 14939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:32.530012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:35679)
0926 20:18:32.530525 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:32.530957 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":113,"threads_started":1}
W20260926 20:18:32.532233 14937 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:38497 (0 consecutive failures): Network error: Failed to ping master at 127.31.119.254:38497: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
I20260926 20:18:32.532917 14939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:32.532705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:41845)
0926 20:18:32.532811 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:32.532861 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:18:32.533596 14939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:32.533414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:58963)
0926 20:18:32.533509 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:32.533550 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:18:32.533670 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 14806
W20260926 20:18:32.533844 14937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:18:32.534211 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:46403
--local_ip_for_outbound_sockets=127.31.119.195
--tserver_master_addrs=127.31.119.254:38497
--webserver_port=37551
--webserver_interface=127.31.119.195
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:32.657457 14940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:32.657758 14940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:32.657805 14940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:32.662212 14940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:32.662292 14940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:32.662353 14940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:18:32.668021 14940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:46403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=37551
--tserver_master_addrs=127.31.119.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:32.669246 14940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:32.670405 14940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:32.680806 14946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:32.680962 14940 server_base.cc:1034] running on GCE node
W20260926 20:18:32.680799 14945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:32.680806 14948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:32.681625 14940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:32.682339 14940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:32.683521 14940 hybrid_clock.cc:648] HybridClock initialized: now 1790453912683507 us; error 30 us; skew 500 ppm
Sep 26 20:18:32 dist-test-slave-vnbh krb5kdc[13495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:32.686513 14940 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:18:32.687481 14940 webserver.cc:460] Webserver started at http://127.31.119.195:37551/ using document root <none> and password file <none>
I20260926 20:18:32.688026 14940 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:32.688088 14940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:32.692204 14940 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:18:32.694541 14955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:32.696008 14940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:18:32.696107 14940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "43e93296fd254148a3023e132d7ee77a"
format_stamp: "Formatted at 2026-09-26 20:18:30 on dist-test-slave-vnbh"
server_key: "9fc8bfd6f0127c3829872e9fab8be4cd"
server_key_iv: "538a1f488d31eb365bcfc21d285a70a9"
server_key_version: "encryptionkey@0"
I20260926 20:18:32.696647 14940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:32.709926 14940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:32.713605 14940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:32.713774 14940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:32.714346 14940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:32.715176 14940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:32.715225 14940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:32.715268 14940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:32.715284 14940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:32.726608 14940 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:46403
I20260926 20:18:32.726634 15068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:46403 every 8 connection(s)
I20260926 20:18:32.727759 14940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:18:32.731884 15071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:32.730210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:38877)
0926 20:18:32.730713 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:32.731151 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":127,"threads_started":1}
W20260926 20:18:32.732385 15069 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:38497 (0 consecutive failures): Network error: Failed to ping master at 127.31.119.254:38497: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
I20260926 20:18:32.732599 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 14940
I20260926 20:18:32.733227 15071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:32.732830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:42255)
0926 20:18:32.733044 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:32.733154 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260926 20:18:32.734052 15071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:32.733845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:50637)
0926 20:18:32.733941 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:32.733997 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:18:32.734251 15069 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:18:33.395457 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:33.394997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:52761)
0926 20:18:33.395264 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:33.395338 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260926 20:18:33.489111 15074 raft_consensus.cc:482] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:18:33.489372 15074 raft_consensus.cc:504] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:33.490576 15074 leader_election.cc:288] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057), 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403)
Sep 26 20:18:33 dist-test-slave-vnbh krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.195@KRBTEST.COM
Sep 26 20:18:33 dist-test-slave-vnbh krb5kdc[13495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453912, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.194@KRBTEST.COM
I20260926 20:18:33.509604 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:33.490783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:33057 (local address 127.31.119.193:37867)
0926 20:18:33.490979 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:33.490995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:33.491115 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:33.493419 (+ 2304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:33.493424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:33.494056 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:33.494925 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:33.494938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:33.497000 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:33.497004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:33.498855 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:33.498869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:33.500072 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:33.500076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:33.500437 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:33.501661 (+ 1224us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:33.501680 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:33.504324 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:33.506836 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:33.506843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:33.506859 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:33.507264 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:33.507668 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:33.507671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:33.507674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:33.507841 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:33.508628 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:33.508631 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:33.508873 (+ 242us) client_negotiation.cc:769] Sending connection context
0926 20:18:33.509050 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 20:18:33.509461 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":368,"spinlock_wait_cycles":21760}
I20260926 20:18:33.509776 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:33.490921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:46403 (local address 127.31.119.193:45595)
0926 20:18:33.491290 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:33.491306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:33.491408 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:33.493653 (+ 2245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:33.493657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:33.494056 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:33.494916 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:33.494938 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:33.497000 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:33.497004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:33.498855 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:33.498869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:33.500072 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:33.500076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:33.500439 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:33.501644 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:33.501671 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:33.503875 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:33.506866 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:33.506871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:33.506873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:33.507278 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:33.507702 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:33.507706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:33.507708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:33.507843 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:33.508565 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:33.508573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:33.508881 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 20:18:33.509050 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 20:18:33.509549 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":691,"spinlock_wait_cycles":76672,"thread_start_us":91,"threads_started":1}
I20260926 20:18:33.509940 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:33.491045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:45595 (local address 127.31.119.195:46403)
0926 20:18:33.491471 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:33.491482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:33.491512 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:18:33.492367 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:33.492376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:33.492882 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:33.493228 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:33.495150 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:33.496795 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:33.499040 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:33.499906 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:33.500116 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:33.504224 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:33.504231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:33.504247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:33.504283 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:33.506704 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:33.507439 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:33.507444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:33.507449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:33.507585 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:33.507967 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:33.507971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:33.507973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:33.508419 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:33.508579 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:33.509402 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 20:18:33.509724 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":109,"threads_started":1}
I20260926 20:18:33.510573 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:33.491033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:37867 (local address 127.31.119.194:33057)
0926 20:18:33.491383 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:33.491401 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:33.491446 (+ 45us) server_negotiation.cc:407] Connection header received
0926 20:18:33.492334 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:33.492344 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:33.492882 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:33.493228 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:33.495150 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:33.496795 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:33.499040 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:33.499906 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:33.500116 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:33.504586 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:33.504594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:33.504614 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:33.504657 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:33.506672 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:33.507415 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:33.507421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:33.507426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:33.507555 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:33.507948 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:33.507951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:33.507953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:33.508447 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:33.508579 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:33.510197 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0926 20:18:33.510448 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":64,"threads_started":1}
W20260926 20:18:33.512984 14692 leader_election.cc:341] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403): Not found: Tablet not found: 5e55f6773bd140faa9a6b60900643be9
W20260926 20:18:33.513201 14689 leader_election.cc:341] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057): Not found: Tablet not found: 5e55f6773bd140faa9a6b60900643be9
I20260926 20:18:33.513365 14689 leader_election.cc:302] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f85a54b3d5432f8c2089249cc619a7; no voters: 43e93296fd254148a3023e132d7ee77a, d08241111ed74731884a8c87ebcc0ed5
I20260926 20:18:33.513624 15074 raft_consensus.cc:2726] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:18:33.534895 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:33.534479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:41319)
0926 20:18:33.534728 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:33.534787 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260926 20:18:33.735303 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:33.734848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:46505)
0926 20:18:33.735133 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:33.735194 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
I20260926 20:18:34.396827 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:34.396361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:34655)
0926 20:18:34.396664 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:34.396728 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
I20260926 20:18:34.536129 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:34.535694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:42573)
0926 20:18:34.535958 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:34.536016 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260926 20:18:34.736569 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:34.736097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:52559)
0926 20:18:34.736374 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:34.736454 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260926 20:18:35.181972 15083 raft_consensus.cc:482] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:18:35.182143 15083 raft_consensus.cc:504] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:35.182646 15083 leader_election.cc:288] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057), 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403)
W20260926 20:18:35.183569 14692 leader_election.cc:341] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403): Not found: Tablet not found: 5e55f6773bd140faa9a6b60900643be9
W20260926 20:18:35.183666 14689 leader_election.cc:341] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057): Not found: Tablet not found: 5e55f6773bd140faa9a6b60900643be9
I20260926 20:18:35.183763 14689 leader_election.cc:302] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f85a54b3d5432f8c2089249cc619a7; no voters: 43e93296fd254148a3023e132d7ee77a, d08241111ed74731884a8c87ebcc0ed5
I20260926 20:18:35.183969 15083 raft_consensus.cc:2726] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:18:35.398455 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:35.397953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:35369)
0926 20:18:35.398261 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:35.398332 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260926 20:18:35.537503 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:35.536936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:35525)
0926 20:18:35.537235 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:35.537403 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
I20260926 20:18:35.738217 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:35.737704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:46827)
0926 20:18:35.738031 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:35.738111 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":122,"threads_started":1}
I20260926 20:18:36.399971 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:36.399440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:40757)
0926 20:18:36.399728 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:36.399817 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
I20260926 20:18:36.538755 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:36.538312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:56435)
0926 20:18:36.538585 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:36.538657 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
I20260926 20:18:36.739513 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:36.739068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:33053)
0926 20:18:36.739360 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:36.739418 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":120,"threads_started":1}
I20260926 20:18:37.401708 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:37.401132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:56799)
0926 20:18:37.401530 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:37.401596 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":141,"threads_started":1}
I20260926 20:18:37.540187 15091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:37.539710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:58823)
0926 20:18:37.540034 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:37.540090 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":92,"threads_started":1}
I20260926 20:18:37.740808 15092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:37.740366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:55977)
0926 20:18:37.740653 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:37.740711 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
I20260926 20:18:38.135809 15093 raft_consensus.cc:482] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:18:38.135995 15093 raft_consensus.cc:504] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:38.136503 15093 leader_election.cc:288] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057), 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403)
W20260926 20:18:38.137516 14689 leader_election.cc:341] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057): Not found: Tablet not found: 5e55f6773bd140faa9a6b60900643be9
W20260926 20:18:38.137645 14692 leader_election.cc:341] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403): Not found: Tablet not found: 5e55f6773bd140faa9a6b60900643be9
I20260926 20:18:38.137760 14692 leader_election.cc:302] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f85a54b3d5432f8c2089249cc619a7; no voters: 43e93296fd254148a3023e132d7ee77a, d08241111ed74731884a8c87ebcc0ed5
I20260926 20:18:38.138015 15093 raft_consensus.cc:2726] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:18:38.403142 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:38.402632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:49695)
0926 20:18:38.402980 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:38.403042 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":83,"threads_started":1}
I20260926 20:18:38.541656 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:38.541090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:43803)
0926 20:18:38.541489 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:38.541552 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":120,"threads_started":1}
I20260926 20:18:38.742388 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:38.741929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:51189)
0926 20:18:38.742192 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:38.742251 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260926 20:18:39.404608 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:39.404104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:56343)
0926 20:18:39.404433 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:39.404504 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20260926 20:18:39.543118 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:39.542617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:47655)
0926 20:18:39.542948 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:39.543018 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":82,"threads_started":1}
I20260926 20:18:39.743790 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:39.743275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:35463)
0926 20:18:39.743617 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:39.743680 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260926 20:18:40.406054 15100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:40.405587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:53471)
0926 20:18:40.405870 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:40.405926 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":120,"threads_started":1}
I20260926 20:18:40.544523 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:40.544055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:52249)
0926 20:18:40.544351 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:40.544424 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260926 20:18:40.745180 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:40.744760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:56121)
0926 20:18:40.745018 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:40.745083 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260926 20:18:41.329411 15103 raft_consensus.cc:482] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:18:41.329542 15103 raft_consensus.cc:504] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:41.330024 15103 leader_election.cc:288] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057), 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403)
W20260926 20:18:41.330842 14689 leader_election.cc:341] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057): Not found: Tablet not found: 5e55f6773bd140faa9a6b60900643be9
W20260926 20:18:41.330926 14692 leader_election.cc:341] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403): Not found: Tablet not found: 5e55f6773bd140faa9a6b60900643be9
I20260926 20:18:41.331053 14692 leader_election.cc:302] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f85a54b3d5432f8c2089249cc619a7; no voters: 43e93296fd254148a3023e132d7ee77a, d08241111ed74731884a8c87ebcc0ed5
I20260926 20:18:41.331223 15103 raft_consensus.cc:2726] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:18:41.407495 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:41.407006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:55705)
0926 20:18:41.407301 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:41.407362 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
I20260926 20:18:41.546052 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:41.545530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:38913)
0926 20:18:41.545865 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:41.545938 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":78,"threads_started":1}
I20260926 20:18:41.746568 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:41.746065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:51897)
0926 20:18:41.746404 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:41.746473 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
I20260926 20:18:42.408988 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:42.408533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:39473)
0926 20:18:42.408807 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:42.408873 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260926 20:18:42.547641 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:42.547101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:48847)
0926 20:18:42.547419 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:42.547494 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260926 20:18:42.747978 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:42.747501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:34791)
0926 20:18:42.747807 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:42.747872 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260926 20:18:43.410319 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:43.409800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:47231)
0926 20:18:43.410125 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:43.410192 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260926 20:18:43.548902 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:43.548424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:56537)
0926 20:18:43.548705 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:43.548759 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260926 20:18:43.749550 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:43.748926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:59937)
0926 20:18:43.749255 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:43.749443 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":90,"threads_started":1}
I20260926 20:18:44.411664 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:44.411208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:36569)
0926 20:18:44.411499 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:44.411562 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":77,"threads_started":1}
I20260926 20:18:44.550429 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:44.549855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:38481)
0926 20:18:44.550206 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:44.550273 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260926 20:18:44.750838 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:44.750390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:33803)
0926 20:18:44.750667 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:44.750725 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260926 20:18:45.413025 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:45.412531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:55131)
0926 20:18:45.412841 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:45.412923 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
I20260926 20:18:45.551735 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:45.551292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:50801)
0926 20:18:45.551582 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:45.551638 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
I20260926 20:18:45.752235 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:45.751763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:51501)
0926 20:18:45.752073 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:45.752134 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":133,"threads_started":1}
I20260926 20:18:46.414287 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:46.413871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:45493)
0926 20:18:46.414115 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:46.414173 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260926 20:18:46.552978 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:46.552562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:39253)
0926 20:18:46.552818 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:46.552880 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260926 20:18:46.700063 15121 raft_consensus.cc:482] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:18:46.700269 15121 raft_consensus.cc:504] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08241111ed74731884a8c87ebcc0ed5" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 33057 } } peers { permanent_uuid: "43e93296fd254148a3023e132d7ee77a" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 46403 } } peers { permanent_uuid: "a5f85a54b3d5432f8c2089249cc619a7" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 44169 } }
I20260926 20:18:46.700678 15121 leader_election.cc:288] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057), 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403)
W20260926 20:18:46.701547 14692 leader_election.cc:341] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43e93296fd254148a3023e132d7ee77a (127.31.119.195:46403): Not found: Tablet not found: 5e55f6773bd140faa9a6b60900643be9
W20260926 20:18:46.701637 14689 leader_election.cc:341] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d08241111ed74731884a8c87ebcc0ed5 (127.31.119.194:33057): Not found: Tablet not found: 5e55f6773bd140faa9a6b60900643be9
I20260926 20:18:46.701737 14689 leader_election.cc:302] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5f85a54b3d5432f8c2089249cc619a7; no voters: 43e93296fd254148a3023e132d7ee77a, d08241111ed74731884a8c87ebcc0ed5
I20260926 20:18:46.701875 15121 raft_consensus.cc:2726] T 5e55f6773bd140faa9a6b60900643be9 P a5f85a54b3d5432f8c2089249cc619a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:18:46.753741 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:46.753193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.195:54697)
0926 20:18:46.753540 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:46.753605 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260926 20:18:47.415970 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:47.415472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.193:39661)
0926 20:18:47.415765 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:47.415827 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260926 20:18:47.554275 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:47.553865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38497 (local address 127.31.119.194:58997)
0926 20:18:47.554114 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:47.554170 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260926 20:18:47.741485 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 14639
I20260926 20:18:47.748085 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 14806
I20260926 20:18:47.754215 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 14940
2026-09-26T20:18:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19595 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:18:47 dist-test-slave-vnbh krb5kdc[15127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:18:47 dist-test-slave-vnbh krb5kdc[15127](info): set up 2 sockets
Sep 26 20:18:47 dist-test-slave-vnbh krb5kdc[15127](info): commencing operation
krb5kdc: starting...
W20260926 20:18:48.806495 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:18:48 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:18:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:18:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:18:48.933490 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38431
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:38431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:49.048238 15140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:49.048535 15140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:49.048578 15140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:49.052654 15140 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:18:49.052721 15140 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:18:49.052737 15140 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:18:49.052753 15140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:18:49.052774 15140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:49.052793 15140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:49.052809 15140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:18:49.052824 15140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:18:49.058271 15140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:38431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:49.059345 15140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:49.060386 15140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:49.070001 15146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:49.070067 15148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:49.070200 15140 server_base.cc:1034] running on GCE node
W20260926 20:18:49.070044 15145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:49.070842 15140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:49.072055 15140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:49.073269 15140 hybrid_clock.cc:648] HybridClock initialized: now 1790453929073240 us; error 47 us; skew 500 ppm
Sep 26 20:18:49 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:49.076576 15140 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:18:49.077840 15140 webserver.cc:460] Webserver started at http://127.31.119.254:38681/ using document root <none> and password file <none>
I20260926 20:18:49.078469 15140 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:49.078567 15140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:49.078819 15140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:49.080982 15140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "96222afb6d4b4488a66622b4db09b67c"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "e26745b6c6ab17aa1abd2e6ea2ea346d"
server_key_iv: "658a832f0d65180bada713482d7dbf45"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.081604 15140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "96222afb6d4b4488a66622b4db09b67c"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "e26745b6c6ab17aa1abd2e6ea2ea346d"
server_key_iv: "658a832f0d65180bada713482d7dbf45"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.085624 15140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:18:49.088280 15155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:49.090163 15140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:49.090322 15140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "96222afb6d4b4488a66622b4db09b67c"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "e26745b6c6ab17aa1abd2e6ea2ea346d"
server_key_iv: "658a832f0d65180bada713482d7dbf45"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.090446 15140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:49.109612 15140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:49.113076 15140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:49.113374 15140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:49.120960 15140 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:38431
I20260926 20:18:49.121274 15207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:38431 every 8 connection(s)
I20260926 20:18:49.122038 15140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:18:49.125052 15208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:49.130415 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 15140
I20260926 20:18:49.130551 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:18:49.130859 32223 external_mini_cluster.cc:1428] Setting key c84d6f9cec813d803097044488c01e47
I20260926 20:18:49.131168 15208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Bootstrap starting.
I20260926 20:18:49.133961 15208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:49.134989 15208 log.cc:824] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:49.137312 15208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: No bootstrap required, opened a new log
Sep 26 20:18:49 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:49.142273 15208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:18:49.142580 15208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:49.142643 15208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96222afb6d4b4488a66622b4db09b67c, State: Initialized, Role: FOLLOWER
I20260926 20:18:49.143131 15208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:18:49.143308 15208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:18:49.143392 15208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:18:49.143523 15208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:49.144906 15208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:18:49.145370 15208 leader_election.cc:302] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96222afb6d4b4488a66622b4db09b67c; no voters:
I20260926 20:18:49.145777 15208 leader_election.cc:288] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:18:49.145849 15213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:18:49.146121 15213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 1 LEADER]: Becoming Leader. State: Replica: 96222afb6d4b4488a66622b4db09b67c, State: Running, Role: LEADER
I20260926 20:18:49.146513 15213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:18:49.147310 15208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:18:49.147624 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.132344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.31.119.254:38431)
0926 20:18:49.132792 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:49.132808 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:49.132846 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:18:49.133613 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:49.133632 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:49.133996 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:49.134344 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:49.135334 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.136385 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.137579 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.138130 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.138262 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.141636 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:49.141642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:49.141660 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:49.141695 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:49.143852 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:49.144457 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:49.144464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:49.144469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:49.144605 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:49.145022 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:49.145025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:49.145026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:49.145687 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:49.145930 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:49.146291 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 20:18:49.146657 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":134,"threads_started":1}
I20260926 20:18:49.147912 15213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96222afb6d4b4488a66622b4db09b67c. Latest consensus state: current_term: 1 leader_uuid: "96222afb6d4b4488a66622b4db09b67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } } }
I20260926 20:18:49.148034 15213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:49.148311 15213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96222afb6d4b4488a66622b4db09b67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } } }
I20260926 20:18:49.148481 15213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: This master's current role is: LEADER
I20260926 20:18:49.148789 15218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:18:49.152213 15218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:18:49.158406 15218 catalog_manager.cc:1347] Generated new cluster ID: 92a0e30593eb40ccbf0d0f25e9bcde74
I20260926 20:18:49.158504 15218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:18:49.168900 15218 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:18:49.170243 15218 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:18:49.177728 15218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Generated new TSK 0
I20260926 20:18:49.178717 15218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:18:49.216575 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38431
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:49.333665 15234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:49.333952 15234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:49.333997 15234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:49.338223 15234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:49.338296 15234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:49.338356 15234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:18:49.343927 15234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:49.345077 15234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:49.346189 15234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:49.356415 15242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:49.356460 15240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:49.356429 15239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:49.356680 15234 server_base.cc:1034] running on GCE node
I20260926 20:18:49.357092 15234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:49.357812 15234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:49.358996 15234 hybrid_clock.cc:648] HybridClock initialized: now 1790453929358977 us; error 31 us; skew 500 ppm
Sep 26 20:18:49 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:49.361814 15234 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:18:49.362788 15234 webserver.cc:460] Webserver started at http://127.31.119.193:38817/ using document root <none> and password file <none>
I20260926 20:18:49.363456 15234 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:49.363529 15234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:49.363770 15234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:49.365942 15234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "4fe1176021e349a0bb93db3d4d2addf4"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "7b266de02642b96f77f9a57938d44bb9"
server_key_iv: "7d9ca099d816253e039d3f9e06dfb76b"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.366503 15234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "4fe1176021e349a0bb93db3d4d2addf4"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "7b266de02642b96f77f9a57938d44bb9"
server_key_iv: "7d9ca099d816253e039d3f9e06dfb76b"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.370327 15234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:18:49.372710 15249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:49.374122 15234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:18:49.374238 15234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "4fe1176021e349a0bb93db3d4d2addf4"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "7b266de02642b96f77f9a57938d44bb9"
server_key_iv: "7d9ca099d816253e039d3f9e06dfb76b"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.374385 15234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:49.397261 15234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:49.401028 15234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:49.401275 15234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:49.402014 15234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:49.402899 15234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:49.402946 15234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:49.403012 15234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:49.403043 15234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:49.412317 15234 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:33263
I20260926 20:18:49.412415 15362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:33263 every 8 connection(s)
I20260926 20:18:49.413609 15234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:18:49.423638 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 15234
Sep 26 20:18:49 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:49.423789 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:18:49.424136 32223 external_mini_cluster.cc:1428] Setting key 510c47ca0c6893455dd38f5312fe6193
I20260926 20:18:49.428109 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.416000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:37115 (local address 127.31.119.254:38431)
0926 20:18:49.416163 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:49.416167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:49.417390 (+ 1223us) server_negotiation.cc:407] Connection header received
0926 20:18:49.418313 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:49.418317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:49.418373 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:49.418483 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:49.419690 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.420226 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.421415 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.421651 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.421763 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.424340 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:49.424343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:49.424346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:49.424368 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:49.426006 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:49.426520 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:49.426525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:49.426527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:49.426611 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:49.426996 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:49.427000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:49.427002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:49.427192 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:49.427292 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:49.427810 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 20:18:49.427944 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:18:49.429003 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.416463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:37115)
0926 20:18:49.417192 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:49.417235 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:49.418103 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:49.418660 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:49.418667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:49.419046 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:49.419536 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.419556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.420350 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.420353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.421248 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.421258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.421812 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.421816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.421980 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.422590 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:49.422608 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:49.424144 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:49.426122 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:49.426128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:49.426144 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:49.426417 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:49.426705 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:49.426708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:49.426710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:49.426849 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:49.427303 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:49.427312 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:49.427573 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 20:18:49.427770 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:18:49.428133 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":110,"threads_started":1}
I20260926 20:18:49.430338 15363 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38431
I20260926 20:18:49.430615 15363 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:49.431159 15363 heartbeater.cc:511] Master 127.31.119.254:38431 requested a full tablet report, sending...
I20260926 20:18:49.432736 15172 ts_manager.cc:194] Registered new tserver with Master: 4fe1176021e349a0bb93db3d4d2addf4 (127.31.119.193:33263)
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
I20260926 20:18:49.434026 15172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:37115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:18:49.453516 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38431
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:49.567834 15368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:49.568122 15368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:49.568167 15368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:49.572315 15368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:49.572395 15368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:49.572491 15368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:18:49.578181 15368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:49.579475 15368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:49.580628 15368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:18:49.591143 15368 server_base.cc:1034] running on GCE node
W20260926 20:18:49.591084 15373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:49.591105 15374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:49.591076 15376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:49.591830 15368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:49.592540 15368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:49.593729 15368 hybrid_clock.cc:648] HybridClock initialized: now 1790453929593690 us; error 62 us; skew 500 ppm
Sep 26 20:18:49 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:49.596915 15368 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:18:49.597904 15368 webserver.cc:460] Webserver started at http://127.31.119.194:42917/ using document root <none> and password file <none>
I20260926 20:18:49.598580 15368 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:49.598675 15368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:49.598903 15368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:49.601146 15368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "0870f20023fd4606a2abf00100895682"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "5a28a185da305088c6bd085d05417e06"
server_key_iv: "2cc290c572268a86208619d989fc47c5"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.601817 15368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "0870f20023fd4606a2abf00100895682"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "5a28a185da305088c6bd085d05417e06"
server_key_iv: "2cc290c572268a86208619d989fc47c5"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.606020 15368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:18:49.608572 15383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:49.610039 15368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:49.610179 15368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "0870f20023fd4606a2abf00100895682"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "5a28a185da305088c6bd085d05417e06"
server_key_iv: "2cc290c572268a86208619d989fc47c5"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.610301 15368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:49.621771 15368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:49.624857 15368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:49.625065 15368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:49.625721 15368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:49.626549 15368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:49.626613 15368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:49.626681 15368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:49.626729 15368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:49.636075 15368 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:38897
I20260926 20:18:49.636081 15496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:38897 every 8 connection(s)
I20260926 20:18:49.637077 15368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:18:49.640856 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 15368
I20260926 20:18:49.640996 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:18:49.641377 32223 external_mini_cluster.cc:1428] Setting key 70028baff01a7aa2ec9722772f6b542c
Sep 26 20:18:49 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
I20260926 20:18:49.652981 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.639359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:41443 (local address 127.31.119.254:38431)
0926 20:18:49.639543 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:49.639547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:49.640471 (+ 924us) server_negotiation.cc:407] Connection header received
0926 20:18:49.642084 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:49.642088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:49.642147 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:49.642230 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:49.643501 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.644031 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.645227 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.645543 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.645655 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.648527 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:49.648532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:49.648538 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:49.648574 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:49.650400 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:49.651112 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:49.651115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:49.651116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:49.651201 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:49.651654 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:49.651656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:49.651657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:49.651854 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:49.651943 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:49.652617 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 20:18:49.652778 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
I20260926 20:18:49.653941 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.639795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:41443)
0926 20:18:49.640293 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:49.640341 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:49.641850 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:49.642431 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:49.642440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:49.642826 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:49.643342 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.643355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.644159 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.644163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.645094 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.645104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.645714 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.645718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.645903 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.646568 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:49.646585 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:49.648339 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:49.650539 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:49.650548 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:49.650573 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:49.650995 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:49.651330 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:49.651334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:49.651338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:49.651554 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:49.651977 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:49.651983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:49.652252 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 20:18:49.652502 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 20:18:49.652795 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260926 20:18:49.655258 15497 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38431
I20260926 20:18:49.655552 15497 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:49.656298 15497 heartbeater.cc:511] Master 127.31.119.254:38431 requested a full tablet report, sending...
I20260926 20:18:49.657644 15172 ts_manager.cc:194] Registered new tserver with Master: 0870f20023fd4606a2abf00100895682 (127.31.119.194:38897)
I20260926 20:18:49.658639 15172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:41443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:18:49.673647 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38431
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:18:49.790756 15502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:18:49.791045 15502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:18:49.791100 15502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:18:49.795418 15502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:18:49.795495 15502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:18:49.795557 15502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:18:49.801118 15502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:18:49.802307 15502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:18:49.803334 15502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:18:49.813228 15507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:49.813437 15502 server_base.cc:1034] running on GCE node
W20260926 20:18:49.813261 15510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:18:49.813246 15508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:18:49.814015 15502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:18:49.814862 15502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:18:49.816062 15502 hybrid_clock.cc:648] HybridClock initialized: now 1790453929816048 us; error 29 us; skew 500 ppm
Sep 26 20:18:49 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:18:49.819090 15502 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:18:49.820004 15502 webserver.cc:460] Webserver started at http://127.31.119.195:44851/ using document root <none> and password file <none>
I20260926 20:18:49.820580 15502 fs_manager.cc:359] Metadata directory not provided
I20260926 20:18:49.820642 15502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:18:49.820927 15502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:18:49.823092 15502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "5ec250c00ba042a5a8452ab386459bdf"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "e354c76c16a5c09b3cef1155d807b7ca"
server_key_iv: "3c6b61ad468b5956cce6c58b24836ed9"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.823668 15502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "5ec250c00ba042a5a8452ab386459bdf"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "e354c76c16a5c09b3cef1155d807b7ca"
server_key_iv: "3c6b61ad468b5956cce6c58b24836ed9"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.827545 15502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 20:18:49.830121 15517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:49.831478 15502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:18:49.831615 15502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "5ec250c00ba042a5a8452ab386459bdf"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "e354c76c16a5c09b3cef1155d807b7ca"
server_key_iv: "3c6b61ad468b5956cce6c58b24836ed9"
server_key_version: "encryptionkey@0"
I20260926 20:18:49.831740 15502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:18:49.844221 15502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:18:49.847852 15502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:18:49.848067 15502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:18:49.848790 15502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:18:49.849812 15502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:18:49.849879 15502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:49.849949 15502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:18:49.849994 15502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:18:49.859205 15502 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:39271
I20260926 20:18:49.859270 15630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:39271 every 8 connection(s)
I20260926 20:18:49.860172 15502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:18:49.861685 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 15502
I20260926 20:18:49.861929 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:18:49.862267 32223 external_mini_cluster.cc:1428] Setting key c97eed463c8feab116c53b7ff22d9de0
Sep 26 20:18:49 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453929, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:18:49.875185 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.863853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:49649 (local address 127.31.119.254:38431)
0926 20:18:49.864021 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:49.864029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:49.864048 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:18:49.865076 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:49.865079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:49.865132 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:49.865231 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:49.866675 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.867204 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.868374 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.868594 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.868697 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.871604 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:49.871607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:49.871610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:49.871636 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:49.873015 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:49.873598 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:49.873601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:49.873602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:49.873679 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:49.874048 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:49.874051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:49.874052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:49.874248 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:49.874330 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:49.874911 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 20:18:49.875014 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:18:49.876199 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.862772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:49649)
0926 20:18:49.863264 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:49.863320 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:49.864867 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:49.865444 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:49.865452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:49.865880 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:18:49.866539 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.866554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.867335 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.867338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.868244 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.868253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.868751 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.868755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.868922 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.869660 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:18:49.869677 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:18:49.871271 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:18:49.873121 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:49.873125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:49.873137 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:49.873490 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:49.873763 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:18:49.873765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:18:49.873766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:18:49.873925 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:18:49.874381 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:18:49.874386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:18:49.874650 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 20:18:49.874826 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 20:18:49.875132 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":137,"threads_started":1}
I20260926 20:18:49.877401 15631 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38431
I20260926 20:18:49.877667 15631 heartbeater.cc:464] Registering TS with master...
I20260926 20:18:49.878260 15631 heartbeater.cc:511] Master 127.31.119.254:38431 requested a full tablet report, sending...
I20260926 20:18:49.879362 15172 ts_manager.cc:194] Registered new tserver with Master: 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271)
I20260926 20:18:49.879927 15172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:49649
I20260926 20:18:49.888839 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:18:49.899060 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.891041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.31.119.254:38431)
0926 20:18:49.891192 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:49.891195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:49.891233 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:18:49.891348 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:49.891351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:49.891393 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:49.891461 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:18:49.892355 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.892921 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.893888 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.894110 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.894190 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.895729 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:18:49.895732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:18:49.895734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:18:49.895758 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:18:49.897364 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:49.897762 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:49.897765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:49.897766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:49.897896 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:18:49.898128 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:18:49.898130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:18:49.898131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:18:49.898354 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:18:49.898456 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:49.898706 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:18:49.898861 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:18:49.902215 15172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:18:49.905404 15172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:18:49.920806 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.915252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:33263 (local address 127.0.0.1:37222)
0926 20:18:49.916030 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:49.916047 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:49.916123 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:49.916775 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:49.916777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:49.916798 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:49.916876 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.916880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.918196 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.918202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.919349 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.919359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.920185 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.920189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.920301 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.920352 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 20:18:49.920487 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 20:18:49.920621 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":69,"thread_start_us":93,"threads_started":1}
I20260926 20:18:49.920944 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.914533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.31.119.195:39271)
0926 20:18:49.915051 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:49.915062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:49.915664 (+ 602us) server_negotiation.cc:407] Connection header received
0926 20:18:49.915751 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:49.915756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:49.915991 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:49.916161 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:49.916615 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.917853 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.919539 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.920364 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.920495 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.920613 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:49.920702 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 20:18:49.920781 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":93,"threads_started":1}
I20260926 20:18:49.921273 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.914624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38897 (local address 127.0.0.1:50862)
0926 20:18:49.915365 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:49.915381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:49.915518 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:49.916501 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:49.916505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:49.916522 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:49.916632 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.916639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.917762 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.917768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.919391 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.919402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.920873 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.920880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.920975 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.920997 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:18:49.921070 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:18:49.921146 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":135,"spinlock_wait_cycles":2816,"thread_start_us":292,"threads_started":1}
I20260926 20:18:49.921455 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.915674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.31.119.193:33263)
0926 20:18:49.916343 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:49.916363 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:49.916381 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:18:49.916440 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:49.916444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:49.916601 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:49.916745 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:49.916996 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.918074 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.919467 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.920062 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.920198 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.920347 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:49.921168 (+ 821us) server_negotiation.cc:299] Negotiation successful
0926 20:18:49.921266 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":97,"threads_started":1}
I20260926 20:18:49.921774 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.915464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.31.119.194:38897)
0926 20:18:49.915866 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:49.915878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:49.915900 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:18:49.915975 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:49.915980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:49.916198 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:49.916346 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:49.916749 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.917636 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.919539 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.920364 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.920495 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.920613 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:49.921485 (+ 872us) server_negotiation.cc:299] Negotiation successful
0926 20:18:49.921619 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":98,"threads_started":1}
I20260926 20:18:49.920822 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.914453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:39271 (local address 127.0.0.1:60540)
0926 20:18:49.915325 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:49.915380 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:49.915517 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:49.916313 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:49.916319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:49.916342 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:49.916492 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.916499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.917979 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.917982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.919391 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.919402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.920490 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.920495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.920575 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.920595 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:18:49.920662 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:18:49.920720 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":123,"spinlock_wait_cycles":8832,"thread_start_us":98,"threads_started":1}
I20260926 20:18:49.924468 15431 tablet_service.cc:1467] Processing CreateTablet for tablet e473be5656304968a7b69d49b2201563 (DEFAULT_TABLE table=test-table [id=dcee86ddf68741a08664eb4451e10bd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:49.924616 15297 tablet_service.cc:1467] Processing CreateTablet for tablet e473be5656304968a7b69d49b2201563 (DEFAULT_TABLE table=test-table [id=dcee86ddf68741a08664eb4451e10bd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:49.924827 15565 tablet_service.cc:1467] Processing CreateTablet for tablet e473be5656304968a7b69d49b2201563 (DEFAULT_TABLE table=test-table [id=dcee86ddf68741a08664eb4451e10bd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:18:49.925647 15431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e473be5656304968a7b69d49b2201563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:49.925693 15297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e473be5656304968a7b69d49b2201563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:49.925698 15565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e473be5656304968a7b69d49b2201563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:18:49.932240 15648 tablet_bootstrap.cc:492] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: Bootstrap starting.
I20260926 20:18:49.933530 15650 tablet_bootstrap.cc:492] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: Bootstrap starting.
I20260926 20:18:49.933530 15649 tablet_bootstrap.cc:492] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Bootstrap starting.
I20260926 20:18:49.935142 15648 tablet_bootstrap.cc:654] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:49.935633 15650 tablet_bootstrap.cc:654] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:49.935640 15649 tablet_bootstrap.cc:654] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Neither blocks nor log segments found. Creating new log.
I20260926 20:18:49.936013 15648 log.cc:824] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:49.936546 15649 log.cc:824] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:49.936915 15650 log.cc:824] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:18:49.938119 15648 tablet_bootstrap.cc:492] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: No bootstrap required, opened a new log
I20260926 20:18:49.938333 15648 ts_tablet_manager.cc:1397] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:18:49.938427 15649 tablet_bootstrap.cc:492] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: No bootstrap required, opened a new log
I20260926 20:18:49.938627 15649 ts_tablet_manager.cc:1397] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 20:18:49.941841 15648 raft_consensus.cc:348] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:18:49.942155 15648 raft_consensus.cc:374] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:49.942210 15648 raft_consensus.cc:729] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0870f20023fd4606a2abf00100895682, State: Initialized, Role: FOLLOWER
I20260926 20:18:49.942350 15649 raft_consensus.cc:348] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:18:49.942637 15649 raft_consensus.cc:374] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:49.942701 15649 raft_consensus.cc:729] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe1176021e349a0bb93db3d4d2addf4, State: Initialized, Role: FOLLOWER
I20260926 20:18:49.942680 15648 consensus_queue.cc:260] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:18:49.943135 15649 consensus_queue.cc:260] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:18:49.943264 15650 tablet_bootstrap.cc:492] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: No bootstrap required, opened a new log
I20260926 20:18:49.943461 15650 ts_tablet_manager.cc:1397] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260926 20:18:49.943504 15497 heartbeater.cc:503] Master 127.31.119.254:38431 was elected leader, sending a full tablet report...
I20260926 20:18:49.943775 15648 ts_tablet_manager.cc:1428] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:18:49.944139 15649 ts_tablet_manager.cc:1428] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 20:18:49.944449 15363 heartbeater.cc:503] Master 127.31.119.254:38431 was elected leader, sending a full tablet report...
I20260926 20:18:49.947471 15650 raft_consensus.cc:348] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:18:49.947782 15650 raft_consensus.cc:374] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:18:49.947860 15650 raft_consensus.cc:729] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec250c00ba042a5a8452ab386459bdf, State: Initialized, Role: FOLLOWER
I20260926 20:18:49.948339 15650 consensus_queue.cc:260] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:18:49.949082 15631 heartbeater.cc:503] Master 127.31.119.254:38431 was elected leader, sending a full tablet report...
I20260926 20:18:49.949481 15650 ts_tablet_manager.cc:1428] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260926 20:18:49.992997 15654 raft_consensus.cc:482] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:18:49.993266 15654 raft_consensus.cc:504] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:18:49.994767 15654 leader_election.cc:288] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271), 4fe1176021e349a0bb93db3d4d2addf4 (127.31.119.193:33263)
I20260926 20:18:50.000036 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.995284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:54631 (local address 127.31.119.195:39271)
0926 20:18:49.995455 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:49.995460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:49.995477 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:18:49.995530 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:49.995533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:49.995584 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:49.995686 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:49.996007 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.996821 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.998701 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.999329 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.999490 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.999532 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:49.999823 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 20:18:49.999901 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:18:50.000044 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.995344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:51129 (local address 127.31.119.193:33263)
0926 20:18:49.995465 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:49.995472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:49.995640 (+ 168us) server_negotiation.cc:407] Connection header received
0926 20:18:49.995793 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:49.995797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:49.995850 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:49.995916 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:18:49.996160 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.996940 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.998701 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.999336 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.999470 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.999527 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:49.999823 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 20:18:49.999944 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:18:50.000038 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.994946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:39271 (local address 127.31.119.194:54631)
0926 20:18:49.995150 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:49.995168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:49.995253 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:49.995717 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:49.995721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:49.995742 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:49.995873 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.995878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.996963 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.996966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.998547 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.998560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.999473 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.999476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.999657 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.999727 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 20:18:49.999876 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 20:18:49.999934 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":135}
I20260926 20:18:50.000046 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:49.995013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:33263 (local address 127.31.119.194:51129)
0926 20:18:49.995548 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:49.995560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:18:49.995680 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:18:49.995946 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:18:49.995951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:18:49.995980 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:18:49.996069 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.996073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.997136 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.997139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.998547 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:18:49.998560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:18:49.999473 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:18:49.999476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:18:49.999652 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:49.999709 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 20:18:49.999796 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:18:49.999879 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":87,"thread_start_us":155,"threads_started":1}
I20260926 20:18:50.000779 15585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e473be5656304968a7b69d49b2201563" candidate_uuid: "0870f20023fd4606a2abf00100895682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec250c00ba042a5a8452ab386459bdf" is_pre_election: true
I20260926 20:18:50.000779 15317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e473be5656304968a7b69d49b2201563" candidate_uuid: "0870f20023fd4606a2abf00100895682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe1176021e349a0bb93db3d4d2addf4" is_pre_election: true
I20260926 20:18:50.001148 15317 raft_consensus.cc:2454] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0870f20023fd4606a2abf00100895682 in term 0.
I20260926 20:18:50.001148 15585 raft_consensus.cc:2454] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0870f20023fd4606a2abf00100895682 in term 0.
I20260926 20:18:50.001749 15387 leader_election.cc:302] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0870f20023fd4606a2abf00100895682, 4fe1176021e349a0bb93db3d4d2addf4; no voters:
I20260926 20:18:50.002043 15654 raft_consensus.cc:2781] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:18:50.002144 15654 raft_consensus.cc:482] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:18:50.002190 15654 raft_consensus.cc:3037] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:50.003309 15654 raft_consensus.cc:504] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:18:50.003937 15654 leader_election.cc:288] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [CANDIDATE]: Term 1 election: Requested vote from peers 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271), 4fe1176021e349a0bb93db3d4d2addf4 (127.31.119.193:33263)
I20260926 20:18:50.004550 15585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e473be5656304968a7b69d49b2201563" candidate_uuid: "0870f20023fd4606a2abf00100895682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec250c00ba042a5a8452ab386459bdf"
I20260926 20:18:50.004715 15317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e473be5656304968a7b69d49b2201563" candidate_uuid: "0870f20023fd4606a2abf00100895682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe1176021e349a0bb93db3d4d2addf4"
I20260926 20:18:50.004730 15585 raft_consensus.cc:3037] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:50.004887 15317 raft_consensus.cc:3037] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:18:50.006022 15585 raft_consensus.cc:2454] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0870f20023fd4606a2abf00100895682 in term 1.
I20260926 20:18:50.006175 15317 raft_consensus.cc:2454] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0870f20023fd4606a2abf00100895682 in term 1.
I20260926 20:18:50.006486 15385 leader_election.cc:302] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0870f20023fd4606a2abf00100895682, 5ec250c00ba042a5a8452ab386459bdf; no voters:
I20260926 20:18:50.006711 15654 raft_consensus.cc:2781] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:18:50.007045 15654 raft_consensus.cc:686] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 1 LEADER]: Becoming Leader. State: Replica: 0870f20023fd4606a2abf00100895682, State: Running, Role: LEADER
I20260926 20:18:50.007484 15654 consensus_queue.cc:237] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:18:50.011243 15171 catalog_manager.cc:5495] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0870f20023fd4606a2abf00100895682 (127.31.119.194). New cstate: current_term: 1 leader_uuid: "0870f20023fd4606a2abf00100895682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } health_report { overall_health: UNKNOWN } } }
I20260926 20:18:50.024235 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:50.019929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.31.119.194:38897)
0926 20:18:50.020063 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:18:50.020073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:18:50.020089 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:18:50.020212 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:18:50.020218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:18:50.020312 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:18:50.020413 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:18:50.020642 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:50.021210 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:50.022232 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:18:50.022505 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:18:50.022601 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:18:50.023089 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:18:50.023592 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:18:50.023704 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:18:50.024041 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 20:18:50.024109 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:18:50.031174 15317 raft_consensus.cc:1261] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Refusing update from remote peer 0870f20023fd4606a2abf00100895682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:18:50.031174 15585 raft_consensus.cc:1261] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf [term 1 FOLLOWER]: Refusing update from remote peer 0870f20023fd4606a2abf00100895682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:18:50.032109 15654 consensus_queue.cc:1035] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:18:50.032371 15658 consensus_queue.cc:1035] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:18:50.040930 15663 mvcc.cc:205] Tried to move back new op lower bound from 7333699297398185984 to 7333699297314152448. Current Snapshot: MvccSnapshot[applied={T|T < 7333699297398185984}]
I20260926 20:18:50.050132 15171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:18:50.050325 15171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:18:50.052857 15171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Sending DeleteTablet for 3 replicas of tablet e473be5656304968a7b69d49b2201563
I20260926 20:18:50.053797 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 15234
I20260926 20:18:50.053781 15565 tablet_service.cc:1514] Processing DeleteTablet for tablet e473be5656304968a7b69d49b2201563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:50 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:60540
I20260926 20:18:50.054008 15431 tablet_service.cc:1514] Processing DeleteTablet for tablet e473be5656304968a7b69d49b2201563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:50 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:50862
I20260926 20:18:50.054797 15672 tablet_replica.cc:330] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: stopping tablet replica
I20260926 20:18:50.055152 15672 raft_consensus.cc:2229] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:18:50.055531 15672 raft_consensus.cc:2258] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:18:50.055922 15671 tablet_replica.cc:330] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: stopping tablet replica
I20260926 20:18:50.056248 15671 raft_consensus.cc:2229] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:18:50.056569 15671 raft_consensus.cc:2258] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:18:50.058327 15671 ts_tablet_manager.cc:1905] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:18:50.058425 15672 ts_tablet_manager.cc:1905] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:18:50.061249 15671 ts_tablet_manager.cc:1918] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:18:50.061484 15671 log.cc:1196] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/e473be5656304968a7b69d49b2201563
I20260926 20:18:50.061890 15671 ts_tablet_manager.cc:1939] T e473be5656304968a7b69d49b2201563 P 5ec250c00ba042a5a8452ab386459bdf: Deleting consensus metadata
I20260926 20:18:50.061883 15672 ts_tablet_manager.cc:1918] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:18:50.061990 15672 log.cc:1196] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/e473be5656304968a7b69d49b2201563
I20260926 20:18:50.062345 15672 ts_tablet_manager.cc:1939] T e473be5656304968a7b69d49b2201563 P 0870f20023fd4606a2abf00100895682: Deleting consensus metadata
W20260926 20:18:50.062363 15159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:33263 (error 108)
I20260926 20:18:50.063441 15157 catalog_manager.cc:4841] TS 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271): tablet e473be5656304968a7b69d49b2201563 (table test-table [id=dcee86ddf68741a08664eb4451e10bd4]) successfully deleted
I20260926 20:18:50.063673 15159 catalog_manager.cc:4841] TS 0870f20023fd4606a2abf00100895682 (127.31.119.194:38897): tablet e473be5656304968a7b69d49b2201563 (table test-table [id=dcee86ddf68741a08664eb4451e10bd4]) successfully deleted
I20260926 20:18:50.064060 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:18:50.063530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:33263 (local address 127.0.0.1:37232)
0926 20:18:50.063925 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 20:18:50.063986 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316}
W20260926 20:18:50.064265 15159 catalog_manager.cc:4584] TS 4fe1176021e349a0bb93db3d4d2addf4 (127.31.119.193:33263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e473be5656304968a7b69d49b2201563: Network error: Client connection negotiation failed: client connection to 127.31.119.193:33263: connect: Connection refused (error 111)
I20260926 20:18:50.064349 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 15368
I20260926 20:18:50.069957 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 15502
I20260926 20:18:50.076356 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 15140
I20260926 20:19:10.083681 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38431
--webserver_interface=127.31.119.254
--webserver_port=38681
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:38431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:10.204039 15674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:10.204334 15674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:10.204376 15674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:10.208621 15674 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:19:10.208686 15674 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:19:10.208702 15674 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:19:10.208717 15674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:19:10.208737 15674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:10.208809 15674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:10.208837 15674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:19:10.208853 15674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:19:10.214497 15674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:38431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=38681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:10.215610 15674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:10.216651 15674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:10.230024 15681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:10.230024 15680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:10.230234 15683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:10.230269 15674 server_base.cc:1034] running on GCE node
I20260926 20:19:10.231040 15674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:10.232326 15674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:10.233508 15674 hybrid_clock.cc:648] HybridClock initialized: now 1790453950233496 us; error 32 us; skew 500 ppm
Sep 26 20:19:10 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:10.240780 15674 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:19:10.241863 15674 webserver.cc:460] Webserver started at http://127.31.119.254:38681/ using document root <none> and password file <none>
I20260926 20:19:10.242455 15674 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:10.242522 15674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:10.246662 15674 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:19:10.248939 15690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:10.250387 15674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:10.250492 15674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "96222afb6d4b4488a66622b4db09b67c"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "e26745b6c6ab17aa1abd2e6ea2ea346d"
server_key_iv: "658a832f0d65180bada713482d7dbf45"
server_key_version: "encryptionkey@0"
I20260926 20:19:10.251089 15674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:10.265509 15674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:10.268874 15674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:10.269047 15674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:10.276542 15674 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:38431
I20260926 20:19:10.276968 15742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:38431 every 8 connection(s)
I20260926 20:19:10.277864 15674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:19:10.281076 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 15674
I20260926 20:19:10.281595 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:33263
--local_ip_for_outbound_sockets=127.31.119.193
--tserver_master_addrs=127.31.119.254:38431
--webserver_port=38817
--webserver_interface=127.31.119.193
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:19:10.283058 15743 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:19:10.285276 15743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Bootstrap starting.
I20260926 20:19:10.310674 15743 log.cc:824] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:10.321969 15743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:19:10.322448 15743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Bootstrap complete.
I20260926 20:19:10.328394 15743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:19:10.328966 15743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96222afb6d4b4488a66622b4db09b67c, State: Initialized, Role: FOLLOWER
I20260926 20:19:10.329576 15743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:19:10.329753 15743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:19:10.329833 15743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:19:10.329938 15743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:19:10.332746 15743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:19:10.333173 15743 leader_election.cc:302] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96222afb6d4b4488a66622b4db09b67c; no voters:
I20260926 20:19:10.333585 15743 leader_election.cc:288] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:19:10.333772 15747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:19:10.334074 15747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 2 LEADER]: Becoming Leader. State: Replica: 96222afb6d4b4488a66622b4db09b67c, State: Running, Role: LEADER
I20260926 20:19:10.334455 15747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:19:10.334951 15743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:19:10.336113 15749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96222afb6d4b4488a66622b4db09b67c. Latest consensus state: current_term: 2 leader_uuid: "96222afb6d4b4488a66622b4db09b67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } } }
I20260926 20:19:10.336123 15748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "96222afb6d4b4488a66622b4db09b67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } } }
I20260926 20:19:10.336272 15749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:10.336277 15748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:10.336863 15756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:19:10.341848 15756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:19:10.342448 15756 catalog_manager.cc:1259] Loaded cluster ID: 92a0e30593eb40ccbf0d0f25e9bcde74
I20260926 20:19:10.342531 15756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:19:10.344458 15756 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:19:10.345124 15756 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Loaded TSK: 0
I20260926 20:19:10.355973 15756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Generated new TSK 1
I20260926 20:19:10.356302 15756 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:19:10.434430 15745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:10.434749 15745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:10.434851 15745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:10.439584 15745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:10.439685 15745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:10.439795 15745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:19:10.446013 15745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:33263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=38817
--tserver_master_addrs=127.31.119.254:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:10.447187 15745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:10.448259 15745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:10.458740 15773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:10.458750 15770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:10.458734 15771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:10.459088 15745 server_base.cc:1034] running on GCE node
I20260926 20:19:10.459817 15745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:10.460572 15745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:10.461788 15745 hybrid_clock.cc:648] HybridClock initialized: now 1790453950461762 us; error 44 us; skew 500 ppm
Sep 26 20:19:10 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:10.464756 15745 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:19:10.465952 15745 webserver.cc:460] Webserver started at http://127.31.119.193:38817/ using document root <none> and password file <none>
I20260926 20:19:10.466562 15745 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:10.466641 15745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:10.471336 15745 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 20:19:10.473789 15780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:10.475183 15745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:10.475289 15745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "4fe1176021e349a0bb93db3d4d2addf4"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "7b266de02642b96f77f9a57938d44bb9"
server_key_iv: "7d9ca099d816253e039d3f9e06dfb76b"
server_key_version: "encryptionkey@0"
I20260926 20:19:10.475888 15745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:10.486918 15745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:10.490522 15745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:10.490738 15745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:10.491410 15745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:10.492848 15787 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:19:10.497938 15745 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:19:10.498013 15745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 20:19:10.498085 15745 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:19:10.499830 15745 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:19:10.499894 15745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 20:19:10.500160 15787 tablet_bootstrap.cc:492] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Bootstrap starting.
I20260926 20:19:10.511215 15745 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:33263
I20260926 20:19:10.511262 15894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:33263 every 8 connection(s)
I20260926 20:19:10.512593 15745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:19:10.520087 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 15745
I20260926 20:19:10.520604 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:38897
--local_ip_for_outbound_sockets=127.31.119.194
--tserver_master_addrs=127.31.119.254:38431
--webserver_port=42917
--webserver_interface=127.31.119.194
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:19:10.521667 15787 log.cc:824] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:19:10 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:10.530997 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.515035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:49759)
0926 20:19:10.515708 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:10.515751 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:10.516666 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:10.517960 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:10.517968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:10.518427 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:10.518894 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:10.518910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.520463 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.520467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:10.521845 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:10.521857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.522864 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.522869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:10.523062 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.523724 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:10.523741 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:10.525475 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:10.527785 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:10.527791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:10.527804 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:10.528073 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:10.528445 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:10.528449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:10.528451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:10.528591 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:10.529223 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:10.529230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:10.529511 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 20:19:10.529704 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 20:19:10.530054 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":175,"threads_started":1}
I20260926 20:19:10.530984 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.515026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:49759 (local address 127.31.119.254:38431)
0926 20:19:10.515450 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:10.515465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:10.515959 (+ 494us) server_negotiation.cc:407] Connection header received
0926 20:19:10.517138 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:10.517149 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:10.517548 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:10.517905 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:10.519085 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.520298 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.521995 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.522685 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.522906 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.525689 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:10.525695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:10.525708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:10.525738 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:10.527644 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.528199 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.528212 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.528217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.528340 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.528690 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.528695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.528697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.529099 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:10.529241 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:10.529820 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 20:19:10.530109 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":103,"threads_started":1}
I20260926 20:19:10.531610 15787 tablet_bootstrap.cc:492] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:19:10.532127 15787 tablet_bootstrap.cc:492] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Bootstrap complete.
I20260926 20:19:10.533756 15787 ts_tablet_manager.cc:1397] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.007s
I20260926 20:19:10.534009 15895 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38431
I20260926 20:19:10.534250 15895 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:10.534813 15895 heartbeater.cc:511] Master 127.31.119.254:38431 requested a full tablet report, sending...
I20260926 20:19:10.537669 15706 ts_manager.cc:194] Registered new tserver with Master: 4fe1176021e349a0bb93db3d4d2addf4 (127.31.119.193:33263)
I20260926 20:19:10.537269 15787 raft_consensus.cc:348] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:10.537879 15787 raft_consensus.cc:729] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe1176021e349a0bb93db3d4d2addf4, State: Initialized, Role: FOLLOWER
I20260926 20:19:10.538414 15787 consensus_queue.cc:260] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:10.539484 15787 ts_tablet_manager.cc:1428] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:19:10.541983 15706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:49759
I20260926 20:19:10.545024 15895 heartbeater.cc:503] Master 127.31.119.254:38431 was elected leader, sending a full tablet report...
Sep 26 20:19:10 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:19:10.553170 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.541012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:33263 (local address 127.0.0.1:57952)
0926 20:19:10.541703 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:10.541764 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:10.541961 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:10.542635 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:10.542642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:10.543128 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:10.543265 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:10.543273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.544724 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.544728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:10.545914 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:10.545927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.546752 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.546756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:10.546878 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.547494 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:10.547509 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:10.549262 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:10.551166 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:10.551176 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:10.551182 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:10.551502 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:10.551812 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:10.551815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:10.551817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:10.551917 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:10.552488 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:10.552494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:10.552630 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 20:19:10.552769 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 20:19:10.552986 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":142,"threads_started":1}
I20260926 20:19:10.553326 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.541051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.31.119.193:33263)
0926 20:19:10.541388 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:10.541400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:10.541889 (+ 489us) server_negotiation.cc:407] Connection header received
0926 20:19:10.542229 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:10.542236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:10.542460 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:10.542609 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:10.543408 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.544479 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.546057 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.546613 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.546736 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.549448 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:10.549457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:10.549468 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:10.549493 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:10.551045 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.551634 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.551640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.551641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.551724 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.552056 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.552060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.552061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.552370 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:10.552496 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:10.552928 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 20:19:10.553088 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":112,"threads_started":1}
I20260926 20:19:10.554826 15829 tablet_service.cc:1514] Processing DeleteTablet for tablet e473be5656304968a7b69d49b2201563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:18:50 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:57952
I20260926 20:19:10.555862 15904 tablet_replica.cc:330] stopping tablet replica
I20260926 20:19:10.556214 15904 raft_consensus.cc:2229] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:10.556504 15904 raft_consensus.cc:2258] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:10.557670 15904 ts_tablet_manager.cc:1905] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:10.562724 15904 ts_tablet_manager.cc:1918] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:10.562877 15904 log.cc:1196] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal/wals/e473be5656304968a7b69d49b2201563
I20260926 20:19:10.563205 15904 ts_tablet_manager.cc:1939] T e473be5656304968a7b69d49b2201563 P 4fe1176021e349a0bb93db3d4d2addf4: Deleting consensus metadata
I20260926 20:19:10.565855 15694 catalog_manager.cc:4841] TS 4fe1176021e349a0bb93db3d4d2addf4 (127.31.119.193:33263): tablet e473be5656304968a7b69d49b2201563 (table test-table [id=dcee86ddf68741a08664eb4451e10bd4]) successfully deleted
W20260926 20:19:10.650694 15899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:10.651003 15899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:10.651048 15899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:10.655387 15899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:10.655484 15899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:10.655573 15899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:19:10.661109 15899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:38897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=42917
--tserver_master_addrs=127.31.119.254:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:10.662294 15899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:10.663301 15899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:10.673312 15910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:10.673504 15909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:10.673610 15899 server_base.cc:1034] running on GCE node
W20260926 20:19:10.673563 15912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:10.674134 15899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:10.674821 15899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:10.676030 15899 hybrid_clock.cc:648] HybridClock initialized: now 1790453950676008 us; error 40 us; skew 500 ppm
Sep 26 20:19:10 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:10.679054 15899 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:19:10.680146 15899 webserver.cc:460] Webserver started at http://127.31.119.194:42917/ using document root <none> and password file <none>
I20260926 20:19:10.680801 15899 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:10.680920 15899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:10.685376 15899 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:19:10.688108 15919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:10.689790 15899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:10.689960 15899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "0870f20023fd4606a2abf00100895682"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "5a28a185da305088c6bd085d05417e06"
server_key_iv: "2cc290c572268a86208619d989fc47c5"
server_key_version: "encryptionkey@0"
I20260926 20:19:10.690646 15899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:10.729393 15899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:10.732707 15899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:10.732946 15899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:10.733686 15899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:10.734565 15899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:10.734638 15899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:10.734712 15899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:10.734757 15899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:10.744499 15899 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:38897
I20260926 20:19:10.744520 16032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:38897 every 8 connection(s)
I20260926 20:19:10.745671 15899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:19:10.750969 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 15899
I20260926 20:19:10.751435 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:39271
--local_ip_for_outbound_sockets=127.31.119.195
--tserver_master_addrs=127.31.119.254:38431
--webserver_port=44851
--webserver_interface=127.31.119.195
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:19:10 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:10.762086 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.747673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:51123 (local address 127.31.119.254:38431)
0926 20:19:10.747822 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:10.747826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:10.748752 (+ 926us) server_negotiation.cc:407] Connection header received
0926 20:19:10.750194 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:10.750198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:10.750264 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:10.750377 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:10.752048 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.752603 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.753994 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.754283 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.754406 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.757549 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:10.757556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:10.757562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:10.757603 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:10.759422 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.760013 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.760019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.760023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.760130 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.760536 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.760541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.760546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.760847 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:10.760970 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:10.761657 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 20:19:10.761842 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:19:10.762810 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.748033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:51123)
0926 20:19:10.748590 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:10.748634 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:10.749965 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:10.750564 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:10.750572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:10.751117 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:10.751858 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:10.751883 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.752777 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.752779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:10.753850 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:10.753861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.754454 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.754458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:10.754672 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.755373 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:10.755391 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:10.757258 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:10.759562 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:10.759568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:10.759582 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:10.759884 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:10.760253 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:10.760255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:10.760257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:10.760414 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:10.760983 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:10.760988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:10.761239 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 20:19:10.761551 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 20:19:10.761847 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":148,"threads_started":1}
I20260926 20:19:10.765036 16033 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38431
I20260926 20:19:10.765353 16033 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:10.765877 16033 heartbeater.cc:511] Master 127.31.119.254:38431 requested a full tablet report, sending...
I20260926 20:19:10.767028 15706 ts_manager.cc:194] Registered new tserver with Master: 0870f20023fd4606a2abf00100895682 (127.31.119.194:38897)
I20260926 20:19:10.767637 15706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:51123
W20260926 20:19:10.872911 16036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:10.873189 16036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:10.873234 16036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:10.877580 16036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:10.877642 16036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:10.877730 16036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:19:10.883486 16036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:39271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=44851
--tserver_master_addrs=127.31.119.254:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:10.884654 16036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:10.885828 16036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:10.895920 16041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:10.895923 16042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:10.896018 16036 server_base.cc:1034] running on GCE node
W20260926 20:19:10.896086 16044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:10.896574 16036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:10.897235 16036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:10.898460 16036 hybrid_clock.cc:648] HybridClock initialized: now 1790453950898416 us; error 61 us; skew 500 ppm
Sep 26 20:19:10 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:10.901371 16036 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:19:10.902336 16036 webserver.cc:460] Webserver started at http://127.31.119.195:44851/ using document root <none> and password file <none>
I20260926 20:19:10.902926 16036 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:10.903030 16036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:10.907217 16036 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:19:10.909495 16051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:10.911165 16036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:10.911270 16036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "5ec250c00ba042a5a8452ab386459bdf"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "e354c76c16a5c09b3cef1155d807b7ca"
server_key_iv: "3c6b61ad468b5956cce6c58b24836ed9"
server_key_version: "encryptionkey@0"
I20260926 20:19:10.911852 16036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:10.931084 16036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:10.934680 16036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:10.934849 16036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:10.935554 16036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:10.936452 16036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:10.936549 16036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:10.936621 16036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:10.936645 16036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:10.946405 16036 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:39271
I20260926 20:19:10.946429 16164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:39271 every 8 connection(s)
I20260926 20:19:10.947535 16036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:19:10.948829 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 16036
I20260926 20:19:10.959229 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.949694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.31.119.254:38431)
0926 20:19:10.949930 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:10.949935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:10.950240 (+ 305us) server_negotiation.cc:407] Connection header received
0926 20:19:10.950346 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:10.950350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:10.950409 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:10.950520 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:10.951554 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.952098 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.953381 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.953630 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.953718 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.955611 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:10.955617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:10.955623 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:10.955693 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:10.957354 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.957873 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.957877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.957879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.957985 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.958299 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.958302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.958303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.958487 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:10.958651 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:10.958905 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 20:19:10.959032 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 20:19:10 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:10.963809 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.949697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:55957 (local address 127.31.119.254:38431)
0926 20:19:10.949955 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:10.949960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:10.951078 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 20:19:10.952614 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:10.952617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:10.952663 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:10.952740 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:10.954339 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.954925 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.956512 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.956875 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.956991 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.960084 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:10.960088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:10.960090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:10.960121 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:10.961654 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.962197 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.962199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.962201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.962271 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.962710 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.962714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.962715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.962923 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:10.963014 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:10.963569 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 20:19:10.963673 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":205,"thread_start_us":68,"threads_started":1}
I20260926 20:19:10.964850 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.950277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:55957)
0926 20:19:10.950890 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:10.950957 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:10.952373 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:10.952941 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:10.952950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:10.953492 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:10.954189 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:10.954207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.955109 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.955114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:10.956387 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:10.956400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.957041 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.957044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:10.957198 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.957894 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:10.957920 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:10.959839 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:10.961757 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:10.961762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:10.961776 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:10.962107 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:10.962381 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:10.962384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:10.962385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:10.962536 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:10.963038 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:10.963043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:10.963324 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 20:19:10.963497 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 20:19:10.963876 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":141,"threads_started":1}
I20260926 20:19:10.966130 16165 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38431
I20260926 20:19:10.966415 16165 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:10.967007 16165 heartbeater.cc:511] Master 127.31.119.254:38431 requested a full tablet report, sending...
I20260926 20:19:10.968106 15706 ts_manager.cc:194] Registered new tserver with Master: 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271)
I20260926 20:19:10.968665 15706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:55957
I20260926 20:19:10.973510 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:19:10.977633 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.974197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.31.119.254:38431)
0926 20:19:10.974428 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:10.974432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:10.974504 (+ 72us) server_negotiation.cc:407] Connection header received
0926 20:19:10.974657 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:10.974659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:10.974761 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:10.974847 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:19:10.975079 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.975586 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.976441 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.976657 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.976734 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.977097 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:19:10.977235 (+ 138us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 20:19:10.977526 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47434: token expired
Metrics: {"server-negotiator.queue_time_us":149}
W20260926 20:19:10.977720 16168 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47434: token expired
W20260926 20:19:10.978181 15636 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.119.254:38431)
I20260926 20:19:10.985884 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.978579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.31.119.254:38431)
0926 20:19:10.978689 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:10.978692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:10.978750 (+ 58us) server_negotiation.cc:407] Connection header received
0926 20:19:10.978835 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:10.978837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:10.978880 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:10.978942 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:10.979647 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.980207 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.981080 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:10.981361 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:10.981472 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:10.982803 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:10.982807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:10.982809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:10.982835 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:10.984166 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.984594 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.984598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.984599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.984685 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:10.984979 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:10.984983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:10.984985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:10.985166 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:10.985325 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:10.985593 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 20:19:10.985716 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:19:10.988752 15706 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:19:10.992383 15706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:19:11.000833 15829 tablet_service.cc:1467] Processing CreateTablet for tablet 69a289981c0a48bdba78e3f221ad58dd (DEFAULT_TABLE table=test-table [id=5ab471480d53459fb24fc5c62968159a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:11.001564 15829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a289981c0a48bdba78e3f221ad58dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:11.004566 15787 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Bootstrap starting.
I20260926 20:19:11.004716 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.999395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:39271 (local address 127.0.0.1:35084)
0926 20:19:10.999552 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:10.999569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:10.999659 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:11.000666 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:11.000669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:11.000686 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:11.000808 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:11.000815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.002302 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.002305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:11.003244 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:11.003254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.004213 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.004217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:11.004321 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:11.004367 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:19:11.004446 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:19:11.004550 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 20:19:11.005488 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.999857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38897 (local address 127.0.0.1:38846)
0926 20:19:11.000321 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:11.000335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:11.000418 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:11.000987 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:11.000992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:11.001009 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:11.001089 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:11.001094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.002608 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.002612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:11.003678 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:11.003689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.005099 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.005102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:11.005174 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:11.005189 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:19:11.005281 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:19:11.005364 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":114,"threads_started":1}
I20260926 20:19:11.005569 15787 tablet_bootstrap.cc:654] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:11.005610 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:11.000030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38846 (local address 127.31.119.194:38897)
0926 20:19:11.000548 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:11.000560 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:11.000582 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:19:11.000656 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:11.000661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:11.000844 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:11.001022 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:11.001195 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.002474 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.003822 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.004822 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.004952 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:11.005048 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:11.005352 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 20:19:11.005458 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":119,"threads_started":1}
I20260926 20:19:11.006841 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:10.999681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.31.119.195:39271)
0926 20:19:11.000113 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:11.000125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:11.000151 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:19:11.000251 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:11.000258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:11.000512 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:11.000692 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:11.000935 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.002171 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.003375 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.004093 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.004230 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:11.004369 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:11.006547 (+ 2178us) server_negotiation.cc:299] Negotiation successful
0926 20:19:11.006668 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":193,"threads_started":1}
I20260926 20:19:11.007164 15787 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: No bootstrap required, opened a new log
I20260926 20:19:11.007258 15787 ts_tablet_manager.cc:1397] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:19:11.007710 15787 raft_consensus.cc:348] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:11.007838 15787 raft_consensus.cc:374] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:11.007876 15787 raft_consensus.cc:729] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe1176021e349a0bb93db3d4d2addf4, State: Initialized, Role: FOLLOWER
I20260926 20:19:11.008005 15787 consensus_queue.cc:260] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:11.008582 15787 ts_tablet_manager.cc:1428] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:19:11.008956 15967 tablet_service.cc:1467] Processing CreateTablet for tablet 69a289981c0a48bdba78e3f221ad58dd (DEFAULT_TABLE table=test-table [id=5ab471480d53459fb24fc5c62968159a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:11.010054 15967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a289981c0a48bdba78e3f221ad58dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:11.009644 16099 tablet_service.cc:1467] Processing CreateTablet for tablet 69a289981c0a48bdba78e3f221ad58dd (DEFAULT_TABLE table=test-table [id=5ab471480d53459fb24fc5c62968159a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:11.010661 16099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a289981c0a48bdba78e3f221ad58dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 20:19:11.013919 15896 tablet.cc:2369] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:19:11.017387 16178 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Bootstrap starting.
I20260926 20:19:11.019744 16178 tablet_bootstrap.cc:654] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:11.020135 16179 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Bootstrap starting.
I20260926 20:19:11.020713 16178 log.cc:824] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:11.022413 16179 tablet_bootstrap.cc:654] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:11.022645 16178 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: No bootstrap required, opened a new log
I20260926 20:19:11.022854 16178 ts_tablet_manager.cc:1397] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:19:11.023263 16179 log.cc:824] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:11.025225 16179 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: No bootstrap required, opened a new log
I20260926 20:19:11.025465 16179 ts_tablet_manager.cc:1397] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:19:11.026400 16178 raft_consensus.cc:348] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:11.026659 16178 raft_consensus.cc:374] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:11.026731 16178 raft_consensus.cc:729] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec250c00ba042a5a8452ab386459bdf, State: Initialized, Role: FOLLOWER
I20260926 20:19:11.027190 16178 consensus_queue.cc:260] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:11.027902 16165 heartbeater.cc:503] Master 127.31.119.254:38431 was elected leader, sending a full tablet report...
I20260926 20:19:11.028228 16178 ts_tablet_manager.cc:1428] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:19:11.029037 16179 raft_consensus.cc:348] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:11.029362 16179 raft_consensus.cc:374] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:11.029433 16179 raft_consensus.cc:729] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0870f20023fd4606a2abf00100895682, State: Initialized, Role: FOLLOWER
I20260926 20:19:11.029825 16179 consensus_queue.cc:260] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:11.030431 16033 heartbeater.cc:503] Master 127.31.119.254:38431 was elected leader, sending a full tablet report...
I20260926 20:19:11.030800 16179 ts_tablet_manager.cc:1428] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:19:11.075556 16182 raft_consensus.cc:482] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:19:11.075783 16182 raft_consensus.cc:504] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:11.077039 16182 leader_election.cc:288] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0870f20023fd4606a2abf00100895682 (127.31.119.194:38897), 4fe1176021e349a0bb93db3d4d2addf4 (127.31.119.193:33263)
I20260926 20:19:11.081595 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:11.077607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:45801 (local address 127.31.119.194:38897)
0926 20:19:11.077834 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:11.077839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:11.077855 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:19:11.077956 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:11.077959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:11.078013 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:11.078102 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:11.078443 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.079087 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.080355 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.081008 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.081103 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:11.081146 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:11.081427 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 20:19:11.081487 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:19:11.081629 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:11.077504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38897 (local address 127.31.119.195:45801)
0926 20:19:11.077720 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:11.077738 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:11.077855 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:11.078135 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:11.078139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:11.078161 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:11.078315 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:11.078323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.079215 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.079218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:11.080198 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:11.080207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.081140 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.081144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:11.081270 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:11.081322 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 20:19:11.081411 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:19:11.081487 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 20:19:11.082131 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:11.077823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:33263 (local address 127.31.119.195:45601)
0926 20:19:11.078123 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:11.078136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:11.078221 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:11.078570 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:11.078573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:11.078591 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:11.078706 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:11.078712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.079742 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.079744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:11.080752 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:11.080762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.081711 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.081715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:11.081819 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:11.081841 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:19:11.081917 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:19:11.081979 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"spinlock_wait_cycles":896,"thread_start_us":125,"threads_started":1}
I20260926 20:19:11.082401 15987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a289981c0a48bdba78e3f221ad58dd" candidate_uuid: "5ec250c00ba042a5a8452ab386459bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0870f20023fd4606a2abf00100895682" is_pre_election: true
I20260926 20:19:11.082510 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:11.077931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:45601 (local address 127.31.119.193:33263)
0926 20:19:11.078189 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:11.078193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:11.078207 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:19:11.078363 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:11.078369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:11.078452 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:11.078535 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:11.078854 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.079614 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.080892 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.081572 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.082099 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:11.082271 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:11.082310 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 20:19:11.082379 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260926 20:19:11.082743 15987 raft_consensus.cc:2454] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec250c00ba042a5a8452ab386459bdf in term 0.
I20260926 20:19:11.083252 15849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a289981c0a48bdba78e3f221ad58dd" candidate_uuid: "5ec250c00ba042a5a8452ab386459bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe1176021e349a0bb93db3d4d2addf4" is_pre_election: true
I20260926 20:19:11.083357 16055 leader_election.cc:302] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0870f20023fd4606a2abf00100895682, 5ec250c00ba042a5a8452ab386459bdf; no voters:
I20260926 20:19:11.083542 15849 raft_consensus.cc:2454] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec250c00ba042a5a8452ab386459bdf in term 0.
I20260926 20:19:11.083632 16182 raft_consensus.cc:2781] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:19:11.083724 16182 raft_consensus.cc:482] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:19:11.083806 16182 raft_consensus.cc:3037] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:11.084976 16182 raft_consensus.cc:504] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:11.085377 16182 leader_election.cc:288] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [CANDIDATE]: Term 1 election: Requested vote from peers 0870f20023fd4606a2abf00100895682 (127.31.119.194:38897), 4fe1176021e349a0bb93db3d4d2addf4 (127.31.119.193:33263)
I20260926 20:19:11.085752 15987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a289981c0a48bdba78e3f221ad58dd" candidate_uuid: "5ec250c00ba042a5a8452ab386459bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0870f20023fd4606a2abf00100895682"
I20260926 20:19:11.085772 15849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a289981c0a48bdba78e3f221ad58dd" candidate_uuid: "5ec250c00ba042a5a8452ab386459bdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe1176021e349a0bb93db3d4d2addf4"
I20260926 20:19:11.085894 15987 raft_consensus.cc:3037] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:11.085917 15849 raft_consensus.cc:3037] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:11.087257 15849 raft_consensus.cc:2454] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec250c00ba042a5a8452ab386459bdf in term 1.
I20260926 20:19:11.087257 15987 raft_consensus.cc:2454] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec250c00ba042a5a8452ab386459bdf in term 1.
I20260926 20:19:11.087533 16055 leader_election.cc:302] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe1176021e349a0bb93db3d4d2addf4, 5ec250c00ba042a5a8452ab386459bdf; no voters:
I20260926 20:19:11.087715 16182 raft_consensus.cc:2781] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:11.087953 16182 raft_consensus.cc:686] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 1 LEADER]: Becoming Leader. State: Replica: 5ec250c00ba042a5a8452ab386459bdf, State: Running, Role: LEADER
I20260926 20:19:11.088339 16182 consensus_queue.cc:237] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:11.091892 15705 catalog_manager.cc:5495] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195). New cstate: current_term: 1 leader_uuid: "5ec250c00ba042a5a8452ab386459bdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } health_report { overall_health: UNKNOWN } } }
I20260926 20:19:11.116410 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:11.112516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.31.119.195:39271)
0926 20:19:11.112675 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:11.112678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:11.112691 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:19:11.112770 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:11.112772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:11.112836 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:11.112938 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:19:11.113168 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.113690 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.114599 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:11.114885 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:11.114969 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:11.115394 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:19:11.115846 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:19:11.116005 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:11.116122 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 20:19:11.116181 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:19:11.122960 15987 raft_consensus.cc:1261] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Refusing update from remote peer 5ec250c00ba042a5a8452ab386459bdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:11.122994 15849 raft_consensus.cc:1261] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Refusing update from remote peer 5ec250c00ba042a5a8452ab386459bdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:11.123715 16182 consensus_queue.cc:1035] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:11.123916 16186 consensus_queue.cc:1035] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:11.130995 16190 mvcc.cc:205] Tried to move back new op lower bound from 7333699383791587328 to 7333699383660736512. Current Snapshot: MvccSnapshot[applied={T|T < 7333699383791587328}]
I20260926 20:19:11.140231 15707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:11.140413 15707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:11.143275 15707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Sending DeleteTablet for 3 replicas of tablet 69a289981c0a48bdba78e3f221ad58dd
I20260926 20:19:11.144213 15829 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a289981c0a48bdba78e3f221ad58dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:11 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:57952
I20260926 20:19:11.144352 16099 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a289981c0a48bdba78e3f221ad58dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:11 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:35084
I20260926 20:19:11.144949 16200 tablet_replica.cc:330] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: stopping tablet replica
I20260926 20:19:11.145035 15967 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a289981c0a48bdba78e3f221ad58dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:11 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:38846
I20260926 20:19:11.145387 16200 raft_consensus.cc:2229] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:19:11.145641 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 15745
I20260926 20:19:11.146536 16200 raft_consensus.cc:2258] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:11.145645 16201 tablet_replica.cc:330] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: stopping tablet replica
I20260926 20:19:11.150158 16201 raft_consensus.cc:2229] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:11.150591 16201 raft_consensus.cc:2258] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:11.152248 16200 ts_tablet_manager.cc:1905] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:11.153177 16201 ts_tablet_manager.cc:1905] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:19:11.153667 15694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:33263 (error 108)
I20260926 20:19:11.154593 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 15899
I20260926 20:19:11.156291 16200 ts_tablet_manager.cc:1918] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:11.156425 16200 log.cc:1196] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/69a289981c0a48bdba78e3f221ad58dd
I20260926 20:19:11.156787 16200 ts_tablet_manager.cc:1939] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Deleting consensus metadata
I20260926 20:19:11.157766 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:11.157181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:33263 (local address 127.0.0.1:57968)
0926 20:19:11.157555 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:11.157648 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261}
W20260926 20:19:11.158051 15694 catalog_manager.cc:4584] TS 4fe1176021e349a0bb93db3d4d2addf4 (127.31.119.193:33263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69a289981c0a48bdba78e3f221ad58dd: Network error: Client connection negotiation failed: client connection to 127.31.119.193:33263: connect: Connection refused (error 111)
I20260926 20:19:11.160182 15692 catalog_manager.cc:4841] TS 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271): tablet 69a289981c0a48bdba78e3f221ad58dd (table test-table [id=5ab471480d53459fb24fc5c62968159a]) successfully deleted
I20260926 20:19:11.162400 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 16036
I20260926 20:19:11.162550 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:11.162250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38897 (local address 127.0.0.1:38856)
0926 20:19:11.162400 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:11.162462 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.194:38897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 20:19:11.167783 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 15674
I20260926 20:19:31.175242 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38431
--webserver_interface=127.31.119.254
--webserver_port=38681
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:38431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:31.295426 16203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:31.295744 16203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:31.295786 16203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:31.299914 16203 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:19:31.299983 16203 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:19:31.299998 16203 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:19:31.300012 16203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:19:31.300032 16203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:31.300050 16203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:31.300067 16203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:19:31.300083 16203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:19:31.305737 16203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:38431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=38681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:31.306907 16203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:31.307919 16203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:31.318420 16209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:31.318420 16208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:31.318462 16211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:31.318517 16203 server_base.cc:1034] running on GCE node
I20260926 20:19:31.319551 16203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:31.320904 16203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:31.322196 16203 hybrid_clock.cc:648] HybridClock initialized: now 1790453971322151 us; error 78 us; skew 500 ppm
Sep 26 20:19:31 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453971, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:31.331343 16203 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:19:31.332571 16203 webserver.cc:460] Webserver started at http://127.31.119.254:38681/ using document root <none> and password file <none>
I20260926 20:19:31.333217 16203 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:31.333359 16203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:31.337630 16203 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:19:31.339988 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:31.341368 16203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:31.341528 16203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "96222afb6d4b4488a66622b4db09b67c"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "e26745b6c6ab17aa1abd2e6ea2ea346d"
server_key_iv: "658a832f0d65180bada713482d7dbf45"
server_key_version: "encryptionkey@0"
I20260926 20:19:31.342216 16203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:31.356141 16203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:31.359588 16203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:31.359830 16203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:31.367901 16203 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:38431
I20260926 20:19:31.368263 16270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:38431 every 8 connection(s)
I20260926 20:19:31.369105 16203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:19:31.372665 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 16203
I20260926 20:19:31.373165 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:33263
--local_ip_for_outbound_sockets=127.31.119.193
--tserver_master_addrs=127.31.119.254:38431
--webserver_port=38817
--webserver_interface=127.31.119.193
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:19:31.373996 16271 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:19:31.376112 16271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Bootstrap starting.
I20260926 20:19:31.394322 16271 log.cc:824] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:31.407083 16271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:19:31.407712 16271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Bootstrap complete.
I20260926 20:19:31.412565 16271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:19:31.413048 16271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96222afb6d4b4488a66622b4db09b67c, State: Initialized, Role: FOLLOWER
I20260926 20:19:31.413707 16271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:19:31.413849 16271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:19:31.413949 16271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:19:31.414067 16271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:19:31.416415 16271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:19:31.416823 16271 leader_election.cc:302] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96222afb6d4b4488a66622b4db09b67c; no voters:
I20260926 20:19:31.417188 16271 leader_election.cc:288] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:19:31.417379 16275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:19:31.417629 16275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [term 3 LEADER]: Becoming Leader. State: Replica: 96222afb6d4b4488a66622b4db09b67c, State: Running, Role: LEADER
I20260926 20:19:31.417979 16275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } }
I20260926 20:19:31.418620 16271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:19:31.419704 16276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "96222afb6d4b4488a66622b4db09b67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } } }
I20260926 20:19:31.419839 16276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:31.420113 16277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96222afb6d4b4488a66622b4db09b67c. Latest consensus state: current_term: 3 leader_uuid: "96222afb6d4b4488a66622b4db09b67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96222afb6d4b4488a66622b4db09b67c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38431 } } }
I20260926 20:19:31.420192 16277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:31.420629 16284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:19:31.426069 16284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:19:31.426831 16284 catalog_manager.cc:1259] Loaded cluster ID: 92a0e30593eb40ccbf0d0f25e9bcde74
I20260926 20:19:31.426903 16284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:19:31.430013 16284 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:19:31.430904 16284 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Loaded TSK: 0
I20260926 20:19:31.430998 16284 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96222afb6d4b4488a66622b4db09b67c: Loaded TSK: 1
F20260926 20:19:31.431798 16284 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc7abe019ad google::LogMessage::Fail() at ??:0
@ 0x7fc7abe0488a google::LogMessage::SendToLog() at ??:0
@ 0x7fc7abe014de google::LogMessage::Flush() at ??:0
@ 0x7fc7abe02a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc7aeac518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc7aeabdf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc7b676a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc7b674499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc7b6745178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc7b6786df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc7b67a582b std::function<>::operator()() at ??:0
@ 0x7fc7b6744b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc7b6745f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc7b6742579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc7b6786640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc7ae0533d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc7ae053c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc7ae0553f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc7ae044512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc7af6886db start_thread at ??:0
@ 0x7fc7aa8ff61f clone at ??:0
W20260926 20:19:31.518316 16273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:31.518584 16273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:31.518630 16273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:31.523355 16273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:31.523442 16273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:31.523548 16273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:19:31.529906 16273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:33263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=38817
--tserver_master_addrs=127.31.119.254:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:31.531016 16273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:31.532114 16273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:31.542948 16318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:31.543015 16319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:31.542948 16321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:31.543216 16273 server_base.cc:1034] running on GCE node
I20260926 20:19:31.543764 16273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:31.544471 16273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:31.545720 16273 hybrid_clock.cc:648] HybridClock initialized: now 1790453971545708 us; error 48 us; skew 500 ppm
Sep 26 20:19:31 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453971, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:31.548779 16273 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:19:31.549968 16273 webserver.cc:460] Webserver started at http://127.31.119.193:38817/ using document root <none> and password file <none>
I20260926 20:19:31.550582 16273 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:31.550648 16273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:31.555100 16273 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:19:31.557629 16328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:31.559067 16273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:19:31.559221 16273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "4fe1176021e349a0bb93db3d4d2addf4"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "7b266de02642b96f77f9a57938d44bb9"
server_key_iv: "7d9ca099d816253e039d3f9e06dfb76b"
server_key_version: "encryptionkey@0"
I20260926 20:19:31.559811 16273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:31.581048 16273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:31.584780 16273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:31.585038 16273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:31.585809 16273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:31.587020 16335 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:19:31.592971 16273 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:19:31.593110 16273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 20:19:31.593235 16273 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:19:31.595207 16273 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:19:31.595297 16273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 20:19:31.595616 16335 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Bootstrap starting.
I20260926 20:19:31.606468 16273 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:33263
I20260926 20:19:31.606772 16442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:33263 every 8 connection(s)
I20260926 20:19:31.607626 16273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:19:31.611135 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 16273
I20260926 20:19:31.611649 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:38897
--local_ip_for_outbound_sockets=127.31.119.194
--tserver_master_addrs=127.31.119.254:38431
--webserver_port=42917
--webserver_interface=127.31.119.194
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:19:31.621488 16335 log.cc:824] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:31.628746 16335 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:19:31.629181 16335 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Bootstrap complete.
I20260926 20:19:31.630409 16335 ts_tablet_manager.cc:1397] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.004s
I20260926 20:19:31.632887 16335 raft_consensus.cc:348] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:31.633396 16335 raft_consensus.cc:729] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe1176021e349a0bb93db3d4d2addf4, State: Initialized, Role: FOLLOWER
I20260926 20:19:31.633919 16335 consensus_queue.cc:260] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:31.634912 16335 ts_tablet_manager.cc:1428] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260926 20:19:31.723276 16445 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:19:31.609811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:42725)
0926 20:19:31.610338 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:31.610395 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:31.611647 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:31.722159 (+110512us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":156,"threads_started":1}
W20260926 20:19:31.724104 16443 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:38431 (0 consecutive failures): Network error: Failed to ping master at 127.31.119.254:38431: Client connection negotiation failed: client connection to 127.31.119.254:38431: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:19:31.724942 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:31.724616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:41697)
0926 20:19:31.724786 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:31.724870 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 20:19:31.725731 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:31.725485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:46343)
0926 20:19:31.725607 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:31.725666 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 20:19:31.725924 16443 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:19:31.772336 16446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:31.772619 16446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:31.772665 16446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:31.776911 16446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:31.777006 16446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:31.777071 16446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:19:31.782792 16446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:38897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=42917
--tserver_master_addrs=127.31.119.254:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:31.783876 16446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:31.784920 16446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:31.794914 16453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:31.794962 16454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:31.794968 16446 server_base.cc:1034] running on GCE node
W20260926 20:19:31.794914 16456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:31.795651 16446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:31.796284 16446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:31.797402 16446 hybrid_clock.cc:648] HybridClock initialized: now 1790453971797363 us; error 48 us; skew 500 ppm
Sep 26 20:19:31 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453971, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:31.800246 16446 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:19:31.801215 16446 webserver.cc:460] Webserver started at http://127.31.119.194:42917/ using document root <none> and password file <none>
I20260926 20:19:31.801838 16446 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:31.801945 16446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:31.805948 16446 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:31.808320 16463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:31.809725 16446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:31.809867 16446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "0870f20023fd4606a2abf00100895682"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "5a28a185da305088c6bd085d05417e06"
server_key_iv: "2cc290c572268a86208619d989fc47c5"
server_key_version: "encryptionkey@0"
I20260926 20:19:31.810451 16446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:31.825742 16446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:31.829030 16446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:31.829241 16446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:31.829895 16446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:31.830991 16470 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:19:31.835232 16446 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:19:31.835321 16446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 20:19:31.835410 16446 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:19:31.837152 16446 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:19:31.837225 16446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 20:19:31.837534 16470 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Bootstrap starting.
I20260926 20:19:31.843327 16470 log.cc:824] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:31.848016 16446 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:38897
I20260926 20:19:31.848073 16578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:38897 every 8 connection(s)
I20260926 20:19:31.848901 16446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:19:31.851961 16470 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:19:31.852265 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:31.850902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:50573)
0926 20:19:31.851423 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:31.851530 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":157,"threads_started":1}
I20260926 20:19:31.852382 16470 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Bootstrap complete.
I20260926 20:19:31.852761 16470 ts_tablet_manager.cc:1397] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
W20260926 20:19:31.852784 16579 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:38431 (0 consecutive failures): Network error: Failed to ping master at 127.31.119.254:38431: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
I20260926 20:19:31.853211 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 16446
I20260926 20:19:31.853461 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:31.853199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:39467)
0926 20:19:31.853336 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:31.853394 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 20:19:31.853694 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:39271
--local_ip_for_outbound_sockets=127.31.119.195
--tserver_master_addrs=127.31.119.254:38431
--webserver_port=44851
--webserver_interface=127.31.119.195
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:19:31.854120 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:31.853886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:58265)
0926 20:19:31.853993 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:31.854053 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 20:19:31.854290 16579 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:19:31.856703 16470 raft_consensus.cc:348] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:31.857439 16470 raft_consensus.cc:729] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0870f20023fd4606a2abf00100895682, State: Initialized, Role: FOLLOWER
I20260926 20:19:31.858006 16470 consensus_queue.cc:260] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:31.859346 16470 ts_tablet_manager.cc:1428] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260926 20:19:31.859797 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:31.859063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:52401)
0926 20:19:31.859678 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:31.859731 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542}
I20260926 20:19:31.860615 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:31.860410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:44523)
0926 20:19:31.860518 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:31.860564 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 20:19:31.970323 16582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:31.970606 16582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:31.970652 16582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:31.974859 16582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:31.974931 16582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:31.975026 16582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:19:31.980460 16582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:39271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=44851
--tserver_master_addrs=127.31.119.254:38431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:31.981734 16582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:31.982824 16582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:31.993098 16591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:31.993098 16589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:31.993140 16582 server_base.cc:1034] running on GCE node
W20260926 20:19:31.993111 16588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:31.993829 16582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:31.994565 16582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:31.995868 16582 hybrid_clock.cc:648] HybridClock initialized: now 1790453971995843 us; error 39 us; skew 500 ppm
Sep 26 20:19:31 dist-test-slave-vnbh krb5kdc[15127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453971, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:31.999053 16582 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:19:32.000031 16582 webserver.cc:460] Webserver started at http://127.31.119.195:44851/ using document root <none> and password file <none>
I20260926 20:19:32.000684 16582 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:32.000790 16582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:32.005081 16582 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:19:32.009595 16598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:32.011086 16582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:19:32.011268 16582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "5ec250c00ba042a5a8452ab386459bdf"
format_stamp: "Formatted at 2026-09-26 20:18:49 on dist-test-slave-vnbh"
server_key: "e354c76c16a5c09b3cef1155d807b7ca"
server_key_iv: "3c6b61ad468b5956cce6c58b24836ed9"
server_key_version: "encryptionkey@0"
I20260926 20:19:32.011929 16582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:32.034605 16582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:32.038169 16582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:32.038419 16582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:32.039078 16582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:32.039963 16582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:32.040035 16582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:32.040108 16582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:32.040153 16582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:32.049479 16582 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:39271
I20260926 20:19:32.049510 16711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:39271 every 8 connection(s)
I20260926 20:19:32.050541 16582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:19:32.051807 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 16582
I20260926 20:19:32.054809 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:32.053007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:35967)
0926 20:19:32.053579 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:32.054024 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":114,"threads_started":1}
W20260926 20:19:32.055310 16712 heartbeater.cc:644] Failed to heartbeat to 127.31.119.254:38431 (0 consecutive failures): Network error: Failed to ping master at 127.31.119.254:38431: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
I20260926 20:19:32.055893 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:32.055659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:52127)
0926 20:19:32.055789 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:32.055838 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 20:19:32.056537 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:32.056303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:39391)
0926 20:19:32.056448 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:32.056489 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 20:19:32.056742 16712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:19:32.727126 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:32.726628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:41333)
0926 20:19:32.726916 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:32.727025 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":106,"threads_started":1}
I20260926 20:19:32.861923 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:32.861489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:58813)
0926 20:19:32.861760 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:32.861823 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
I20260926 20:19:32.925671 16718 raft_consensus.cc:482] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:19:32.925956 16718 raft_consensus.cc:504] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:32.927146 16718 leader_election.cc:288] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271), 0870f20023fd4606a2abf00100895682 (127.31.119.194:38897)
Sep 26 20:19:32 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453971, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.194@KRBTEST.COM
Sep 26 20:19:32 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453971, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.195@KRBTEST.COM
I20260926 20:19:32.942836 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:32.927360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:39271 (local address 127.31.119.193:56429)
0926 20:19:32.927785 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:32.927802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:32.927922 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:32.929509 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:32.929511 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:32.929753 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:32.930299 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:32.930314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:32.931690 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:32.931692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:32.932703 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:32.932716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:32.933739 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:32.933743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:32.934016 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:32.934769 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:32.934791 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:32.937170 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:32.939468 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:32.939475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:32.939490 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:32.940050 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:32.940511 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:32.940515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:32.940517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:32.940788 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:32.941624 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:32.941631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:32.942016 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 20:19:32.942245 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 20:19:32.942675 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":355,"spinlock_wait_cycles":26112}
I20260926 20:19:32.942893 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:32.927602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:41537 (local address 127.31.119.194:38897)
0926 20:19:32.927920 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:32.927936 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:32.928066 (+ 130us) server_negotiation.cc:407] Connection header received
0926 20:19:32.928358 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:32.928365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:32.928715 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:32.928961 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:32.930469 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:32.931474 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:32.932883 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:32.933582 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:32.933750 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:32.937081 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:32.937087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:32.937100 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:32.937129 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:32.939311 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:32.940179 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:32.940185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:32.940191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:32.940385 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:32.940901 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:32.940905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:32.940907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:32.941499 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:32.941629 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:32.942410 (+ 781us) server_negotiation.cc:299] Negotiation successful
0926 20:19:32.942692 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260926 20:19:32.942936 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:32.927572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:56429 (local address 127.31.119.195:39271)
0926 20:19:32.928037 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:32.928053 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:32.928097 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:19:32.928801 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:32.928807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:32.929234 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:32.929581 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:32.930515 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:32.931529 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:32.932835 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:32.933582 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:32.933750 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:32.937408 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:32.937415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:32.937431 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:32.937468 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:32.939311 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:32.940179 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:32.940185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:32.940191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:32.940357 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:32.940901 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:32.940905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:32.940907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:32.941499 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:32.941629 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:32.942392 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 20:19:32.942692 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":105,"threads_started":1}
I20260926 20:19:32.942835 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:32.927460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:38897 (local address 127.31.119.193:41537)
0926 20:19:32.927939 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:32.927955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:32.928069 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:32.929089 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:32.929095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:32.929757 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:32.930300 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:32.930315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:32.931630 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:32.931637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:32.932729 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:32.932744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:32.933739 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:32.933743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:32.934020 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:32.934796 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:32.934807 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:32.936818 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:32.939468 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:32.939475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:32.939490 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:32.940050 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:32.940524 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:32.940527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:32.940529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:32.940788 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:32.941624 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:32.941631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:32.942016 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 20:19:32.942245 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 20:19:32.942622 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":261,"spinlock_wait_cycles":56448,"thread_start_us":111,"threads_started":1}
I20260926 20:19:32.945197 16532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a289981c0a48bdba78e3f221ad58dd" candidate_uuid: "4fe1176021e349a0bb93db3d4d2addf4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0870f20023fd4606a2abf00100895682" is_pre_election: true
I20260926 20:19:32.945592 16532 raft_consensus.cc:2454] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe1176021e349a0bb93db3d4d2addf4 in term 1.
W20260926 20:19:32.946367 16330 leader_election.cc:341] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271): Not found: Tablet not found: 69a289981c0a48bdba78e3f221ad58dd
I20260926 20:19:32.946532 16330 leader_election.cc:302] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0870f20023fd4606a2abf00100895682, 4fe1176021e349a0bb93db3d4d2addf4; no voters: 5ec250c00ba042a5a8452ab386459bdf
I20260926 20:19:32.946836 16718 raft_consensus.cc:2781] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:19:32.946964 16718 raft_consensus.cc:482] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:19:32.947031 16718 raft_consensus.cc:3037] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:19:32.949179 16718 raft_consensus.cc:504] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:32.949615 16718 leader_election.cc:288] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [CANDIDATE]: Term 2 election: Requested vote from peers 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271), 0870f20023fd4606a2abf00100895682 (127.31.119.194:38897)
I20260926 20:19:32.950078 16532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a289981c0a48bdba78e3f221ad58dd" candidate_uuid: "4fe1176021e349a0bb93db3d4d2addf4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0870f20023fd4606a2abf00100895682"
I20260926 20:19:32.950234 16532 raft_consensus.cc:3037] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 1 FOLLOWER]: Advancing to term 2
W20260926 20:19:32.950403 16330 leader_election.cc:341] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271): Not found: Tablet not found: 69a289981c0a48bdba78e3f221ad58dd
I20260926 20:19:32.952477 16532 raft_consensus.cc:2454] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe1176021e349a0bb93db3d4d2addf4 in term 2.
I20260926 20:19:32.952886 16332 leader_election.cc:302] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0870f20023fd4606a2abf00100895682, 4fe1176021e349a0bb93db3d4d2addf4; no voters: 5ec250c00ba042a5a8452ab386459bdf
I20260926 20:19:32.953083 16718 raft_consensus.cc:2781] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:19:32.953263 16718 raft_consensus.cc:686] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [term 2 LEADER]: Becoming Leader. State: Replica: 4fe1176021e349a0bb93db3d4d2addf4, State: Running, Role: LEADER
I20260926 20:19:32.953683 16718 consensus_queue.cc:237] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:32.954085 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:32.953843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:60915)
0926 20:19:32.953953 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:32.954012 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 20:19:33.057994 16724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:33.057407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:39357)
0926 20:19:33.057722 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:33.057818 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
W20260926 20:19:33.367687 16330 consensus_peers.cc:487] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 -> Peer 5ec250c00ba042a5a8452ab386459bdf (127.31.119.195:39271): Couldn't send request to peer 5ec250c00ba042a5a8452ab386459bdf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69a289981c0a48bdba78e3f221ad58dd. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:19:33.433518 16532 raft_consensus.cc:1261] T 69a289981c0a48bdba78e3f221ad58dd P 0870f20023fd4606a2abf00100895682 [term 2 FOLLOWER]: Refusing update from remote peer 4fe1176021e349a0bb93db3d4d2addf4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 20:19:33.434293 16718 consensus_queue.cc:1035] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 20:19:33.436546 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:33.436155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:49541)
0926 20:19:33.436394 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:33.436451 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260926 20:19:33.438901 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:33.438572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:57423)
0926 20:19:33.438716 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:33.438814 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 20:19:33.439415 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:33.439160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:33695)
0926 20:19:33.439294 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:33.439353 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:19:33.816191 16730 ts_tablet_manager.cc:927] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Initiating tablet copy from peer 4fe1176021e349a0bb93db3d4d2addf4 (127.31.119.193:33263)
I20260926 20:19:33.816818 16730 tablet_copy_client.cc:316] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: tablet copy: Beginning tablet copy session from remote peer at address 127.31.119.193:33263
Sep 26 20:19:33 dist-test-slave-vnbh krb5kdc[15127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453971, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.193@KRBTEST.COM
I20260926 20:19:33.827966 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:33.817391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:33263 (local address 127.31.119.195:52315)
0926 20:19:33.817674 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:33.817694 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:33.817906 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:33.818343 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:33.818349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:33.818896 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:33.819053 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:33.819060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:33.820166 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:33.820169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:33.820964 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:33.820974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:33.821709 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:33.821712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:33.821808 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:33.822399 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:33.822412 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:33.823897 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:33.826073 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:33.826085 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:33.826092 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:33.826350 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:33.826664 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:33.826668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:33.826671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:33.826762 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:33.827243 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:33.827250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:33.827396 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 20:19:33.827583 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 20:19:33.827775 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":131,"threads_started":1}
I20260926 20:19:33.828003 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:33.817495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:52315 (local address 127.31.119.193:33263)
0926 20:19:33.817752 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:33.817759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:33.817858 (+ 99us) server_negotiation.cc:407] Connection header received
0926 20:19:33.817995 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:33.818000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:33.818188 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:33.818308 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:33.819255 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:33.820050 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:33.821115 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:33.821587 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:33.821691 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:33.824054 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:33.824064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:33.824078 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:33.824102 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:33.825952 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:33.826474 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:33.826484 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:33.826486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:33.826565 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:33.826856 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:33.826860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:33.826862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:33.827124 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:33.827273 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:33.827681 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 20:19:33.827850 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260926 20:19:33.829406 16417 tablet_copy_service.cc:140] P 4fe1176021e349a0bb93db3d4d2addf4: Received BeginTabletCopySession request for tablet 69a289981c0a48bdba78e3f221ad58dd from peer 5ec250c00ba042a5a8452ab386459bdf ({username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:52315)
I20260926 20:19:33.829600 16417 tablet_copy_service.cc:161] P 4fe1176021e349a0bb93db3d4d2addf4: Beginning new tablet copy session on tablet 69a289981c0a48bdba78e3f221ad58dd from peer 5ec250c00ba042a5a8452ab386459bdf at {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:52315: session id = 5ec250c00ba042a5a8452ab386459bdf-69a289981c0a48bdba78e3f221ad58dd
I20260926 20:19:33.831053 16417 tablet_copy_source_session.cc:205] T 69a289981c0a48bdba78e3f221ad58dd P 4fe1176021e349a0bb93db3d4d2addf4: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:19:33.835323 16730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a289981c0a48bdba78e3f221ad58dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:33.842125 16730 tablet_copy_client.cc:791] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: tablet copy: Starting download of 0 data blocks...
I20260926 20:19:33.842357 16730 tablet_copy_client.cc:655] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: tablet copy: Starting download of 1 WAL segments...
I20260926 20:19:33.844260 16730 tablet_copy_client.cc:531] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:19:33.846043 16730 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Bootstrap starting.
I20260926 20:19:33.851433 16730 log.cc:824] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:33.858310 16730 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:19:33.858667 16730 tablet_bootstrap.cc:492] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Bootstrap complete.
I20260926 20:19:33.859010 16730 ts_tablet_manager.cc:1397] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260926 20:19:33.861449 16730 raft_consensus.cc:348] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:33.861738 16730 raft_consensus.cc:729] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec250c00ba042a5a8452ab386459bdf, State: Initialized, Role: FOLLOWER
I20260926 20:19:33.862196 16730 consensus_queue.cc:260] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec250c00ba042a5a8452ab386459bdf" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39271 } } peers { permanent_uuid: "0870f20023fd4606a2abf00100895682" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 38897 } } peers { permanent_uuid: "4fe1176021e349a0bb93db3d4d2addf4" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 33263 } }
I20260926 20:19:33.863413 16730 ts_tablet_manager.cc:1428] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 20:19:33.863541 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:33.863227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:44397)
0926 20:19:33.863400 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:33.863457 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 20:19:33.864246 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:33.864000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:57237)
0926 20:19:33.864136 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:33.864193 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:19:33.864217 16417 tablet_copy_service.cc:342] P 4fe1176021e349a0bb93db3d4d2addf4: Request end of tablet copy session 5ec250c00ba042a5a8452ab386459bdf-69a289981c0a48bdba78e3f221ad58dd received from {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:52315
I20260926 20:19:33.864364 16417 tablet_copy_service.cc:433] P 4fe1176021e349a0bb93db3d4d2addf4: ending tablet copy session 5ec250c00ba042a5a8452ab386459bdf-69a289981c0a48bdba78e3f221ad58dd on tablet 69a289981c0a48bdba78e3f221ad58dd with peer 5ec250c00ba042a5a8452ab386459bdf
I20260926 20:19:34.354126 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:34.353632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:50267)
0926 20:19:34.353948 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:34.354033 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":66,"threads_started":1}
I20260926 20:19:34.354074 16666 raft_consensus.cc:1203] T 69a289981c0a48bdba78e3f221ad58dd P 5ec250c00ba042a5a8452ab386459bdf [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 20:19:34.355032 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:34.354786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:41233)
0926 20:19:34.354908 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:34.354963 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 20:19:34.440593 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:34.440165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:38931)
0926 20:19:34.440431 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:34.440497 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":68,"threads_started":1}
I20260926 20:19:35.355626 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:35.354994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:60177)
0926 20:19:35.355414 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:35.355490 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":75,"threads_started":1}
I20260926 20:19:35.356135 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:35.355765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:59497)
0926 20:19:35.355991 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:35.356044 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260926 20:19:35.442049 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:35.441531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:54303)
0926 20:19:35.441874 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:35.441944 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":88,"threads_started":1}
I20260926 20:19:36.356962 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:36.356525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:44739)
0926 20:19:36.356795 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:36.356866 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":108,"threads_started":1}
I20260926 20:19:36.357432 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:36.356926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:55837)
0926 20:19:36.357245 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:36.357333 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":74,"threads_started":1}
I20260926 20:19:36.443454 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:36.442946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:35841)
0926 20:19:36.443261 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:36.443340 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":86,"threads_started":1}
I20260926 20:19:37.358798 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:37.358274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:33093)
0926 20:19:37.358599 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:37.358676 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
I20260926 20:19:37.358866 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:37.358320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:50809)
0926 20:19:37.358599 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:37.358676 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":63,"threads_started":1}
I20260926 20:19:37.359669 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:37.359420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:43429)
0926 20:19:37.359539 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:37.359600 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 20:19:37.445151 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:37.444615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:45277)
0926 20:19:37.444956 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:37.445023 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":93,"threads_started":1}
I20260926 20:19:38.360220 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:38.359667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:58437)
0926 20:19:38.359998 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:38.360080 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
I20260926 20:19:38.360873 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:38.360443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:45985)
0926 20:19:38.360685 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:38.360741 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
I20260926 20:19:38.446496 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:38.446067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:53285)
0926 20:19:38.446336 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:38.446397 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260926 20:19:39.361655 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:39.360996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:56769)
0926 20:19:39.361459 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:39.361525 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":93,"threads_started":1}
I20260926 20:19:39.361933 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:39.361501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:53723)
0926 20:19:39.361777 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:39.361842 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":66,"threads_started":1}
I20260926 20:19:39.447937 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:39.447437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:51743)
0926 20:19:39.447757 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:39.447827 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":79,"threads_started":1}
I20260926 20:19:40.363126 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:40.362654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:51057)
0926 20:19:40.362974 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:40.363029 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":77,"threads_started":1}
I20260926 20:19:40.363152 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:40.362628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:56129)
0926 20:19:40.362961 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:40.363032 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":94,"threads_started":1}
I20260926 20:19:40.449491 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:40.448847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:36869)
0926 20:19:40.449220 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:40.449351 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":144,"threads_started":1}
I20260926 20:19:41.364396 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:41.363897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:50049)
0926 20:19:41.364202 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:41.364258 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
I20260926 20:19:41.364418 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:41.363902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:60087)
0926 20:19:41.364170 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:41.364272 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260926 20:19:41.450994 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:41.450414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:45707)
0926 20:19:41.450784 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:41.450863 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":97,"threads_started":1}
I20260926 20:19:42.365983 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:42.365473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:55211)
0926 20:19:42.365793 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:42.365862 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":83,"threads_started":1}
I20260926 20:19:42.366039 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:42.365473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:59011)
0926 20:19:42.365848 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:42.365938 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":80,"threads_started":1}
I20260926 20:19:42.452529 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:42.452050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:53153)
0926 20:19:42.452362 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:42.452433 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260926 20:19:43.367296 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:43.366833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:54371)
0926 20:19:43.367126 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:43.367191 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260926 20:19:43.367363 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:43.366831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:53739)
0926 20:19:43.367163 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:43.367252 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260926 20:19:43.454126 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:43.453653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:55661)
0926 20:19:43.453955 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:43.454024 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260926 20:19:44.368665 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:44.368159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:52179)
0926 20:19:44.368485 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:44.368568 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":76,"threads_started":1}
I20260926 20:19:44.368714 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:44.368196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:57177)
0926 20:19:44.368518 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:44.368609 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260926 20:19:44.455524 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:44.455049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:35279)
0926 20:19:44.455359 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:44.455424 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
I20260926 20:19:45.370144 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:45.369693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:35757)
0926 20:19:45.369978 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:45.370046 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260926 20:19:45.370216 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:45.369693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:43201)
0926 20:19:45.370022 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:45.370112 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":119,"threads_started":1}
I20260926 20:19:45.457067 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:45.456533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:45101)
0926 20:19:45.456901 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:45.456966 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":94,"threads_started":1}
I20260926 20:19:46.371644 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:46.371173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.193:44881)
0926 20:19:46.371473 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:46.371547 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":144,"threads_started":1}
I20260926 20:19:46.371694 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:46.371173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.195:38399)
0926 20:19:46.371493 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:46.371553 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":139,"threads_started":1}
I20260926 20:19:46.458424 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:46.457912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38431 (local address 127.31.119.194:40671)
0926 20:19:46.458250 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:46.458316 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.254:38431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":92,"threads_started":1}
I20260926 20:19:47.058485 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 16273
I20260926 20:19:47.064760 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 16446
I20260926 20:19:47.071061 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 16582
2026-09-26T20:19:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59317 ms)
[----------] 30 tests from SecurityITest (249093 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:19:47 dist-test-slave-vnbh krb5kdc[16779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:19:47 dist-test-slave-vnbh krb5kdc[16779](info): set up 2 sockets
Sep 26 20:19:47 dist-test-slave-vnbh krb5kdc[16779](info): commencing operation
krb5kdc: starting...
W20260926 20:19:48.178176 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:19:48 dist-test-slave-vnbh krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:19:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:19:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:19:48.313117 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:39053
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:48.427791 16792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:48.428144 16792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:48.428226 16792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:48.432422 16792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:19:48.432534 16792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:48.432571 16792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:48.432605 16792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:19:48.432632 16792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:19:48.437966 16792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:48.439095 16792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:48.440176 16792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:48.449563 16800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:48.449544 16798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:48.449542 16797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:48.449584 16792 server_base.cc:1034] running on GCE node
I20260926 20:19:48.450376 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:48.451676 16792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:48.452925 16792 hybrid_clock.cc:648] HybridClock initialized: now 1790453988452912 us; error 28 us; skew 500 ppm
Sep 26 20:19:48 dist-test-slave-vnbh krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453988, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:48.461560 16792 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:19:48.462512 16792 webserver.cc:460] Webserver started at http://127.31.119.254:38043/ using document root <none> and password file <none>
I20260926 20:19:48.463091 16792 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:48.463155 16792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:48.463325 16792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:48.465546 16792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "48035303216f40e19c18dd05ab1cbf0c"
format_stamp: "Formatted at 2026-09-26 20:19:48 on dist-test-slave-vnbh"
server_key: "79f44c3feb2c6ec08e702170e8ccfef6"
server_key_iv: "9b62be263e22d860fccfd6b8da7c0b6c"
server_key_version: "encryptionkey@0"
I20260926 20:19:48.466079 16792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "48035303216f40e19c18dd05ab1cbf0c"
format_stamp: "Formatted at 2026-09-26 20:19:48 on dist-test-slave-vnbh"
server_key: "79f44c3feb2c6ec08e702170e8ccfef6"
server_key_iv: "9b62be263e22d860fccfd6b8da7c0b6c"
server_key_version: "encryptionkey@0"
I20260926 20:19:48.469951 16792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:19:48.472589 16807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:48.474110 16792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:48.474222 16792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "48035303216f40e19c18dd05ab1cbf0c"
format_stamp: "Formatted at 2026-09-26 20:19:48 on dist-test-slave-vnbh"
server_key: "79f44c3feb2c6ec08e702170e8ccfef6"
server_key_iv: "9b62be263e22d860fccfd6b8da7c0b6c"
server_key_version: "encryptionkey@0"
I20260926 20:19:48.474401 16792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:48.486847 16792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:48.490528 16792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:48.490747 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:48.498498 16792 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:39053
I20260926 20:19:48.498513 16859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:39053 every 8 connection(s)
I20260926 20:19:48.499568 16792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:19:48.500169 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 16792
I20260926 20:19:48.500310 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:19:48.500623 32223 external_mini_cluster.cc:1428] Setting key 53de6615c10644eaa45a0b5ac2e6d4dc
I20260926 20:19:48.503108 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:48.510500 16860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c: Bootstrap starting.
Sep 26 20:19:48 dist-test-slave-vnbh krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:48.513082 16860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:48.513916 16860 log.cc:824] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:48.516098 16860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c: No bootstrap required, opened a new log
I20260926 20:19:48.516901 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:48.502151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.31.119.254:39053)
0926 20:19:48.502594 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:48.502613 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:48.502659 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:19:48.503425 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:48.503445 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:48.503785 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:48.504083 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:48.505090 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:48.506291 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:48.507597 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:48.508172 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:48.508324 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:48.511494 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:48.511500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:48.511514 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:48.511546 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:48.513725 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:48.514298 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:48.514305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:48.514312 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:48.514449 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:48.514767 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:48.514771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:48.514773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:48.515205 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:48.515427 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:48.515746 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 20:19:48.516061 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":158,"threads_started":1}
I20260926 20:19:48.520470 16860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48035303216f40e19c18dd05ab1cbf0c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 39053 } }
I20260926 20:19:48.520788 16860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:48.520884 16860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48035303216f40e19c18dd05ab1cbf0c, State: Initialized, Role: FOLLOWER
I20260926 20:19:48.521411 16860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48035303216f40e19c18dd05ab1cbf0c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 39053 } }
I20260926 20:19:48.521575 16860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:19:48.521678 16860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:19:48.521799 16860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:48.522840 16860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48035303216f40e19c18dd05ab1cbf0c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 39053 } }
I20260926 20:19:48.523224 16860 leader_election.cc:302] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48035303216f40e19c18dd05ab1cbf0c; no voters:
I20260926 20:19:48.523597 16860 leader_election.cc:288] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:19:48.523857 16865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:48.524147 16865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [term 1 LEADER]: Becoming Leader. State: Replica: 48035303216f40e19c18dd05ab1cbf0c, State: Running, Role: LEADER
I20260926 20:19:48.524471 16865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48035303216f40e19c18dd05ab1cbf0c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 39053 } }
I20260926 20:19:48.525182 16860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:19:48.526540 16867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48035303216f40e19c18dd05ab1cbf0c. Latest consensus state: current_term: 1 leader_uuid: "48035303216f40e19c18dd05ab1cbf0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48035303216f40e19c18dd05ab1cbf0c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 39053 } } }
I20260926 20:19:48.526682 16867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:48.527139 16874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:19:48.527072 16866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48035303216f40e19c18dd05ab1cbf0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48035303216f40e19c18dd05ab1cbf0c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 39053 } } }
I20260926 20:19:48.527730 16866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:48.530269 16874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:19:48.536111 16874 catalog_manager.cc:1347] Generated new cluster ID: 475ee0ed72164987bbd625b7667e0c3e
I20260926 20:19:48.536212 16874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:19:48.558568 16874 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:19:48.559687 16874 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:19:48.569200 16874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c: Generated new TSK 0
I20260926 20:19:48.570227 16874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:19:48.605734 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:39053
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:48.724682 16886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:48.725055 16886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:48.725102 16886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:48.729532 16886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:48.729602 16886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:48.729663 16886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:19:48.735103 16886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:48.736124 16886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:48.737090 16886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:48.747314 16891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:48.747314 16894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:48.747466 16892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:48.747581 16886 server_base.cc:1034] running on GCE node
I20260926 20:19:48.748121 16886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:48.748826 16886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:48.750047 16886 hybrid_clock.cc:648] HybridClock initialized: now 1790453988750018 us; error 49 us; skew 500 ppm
Sep 26 20:19:48 dist-test-slave-vnbh krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453988, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:48.753458 16886 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:19:48.754360 16886 webserver.cc:460] Webserver started at http://127.31.119.193:40171/ using document root <none> and password file <none>
I20260926 20:19:48.755026 16886 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:48.755101 16886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:48.755283 16886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:48.757453 16886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "069a933aab574a7aa008844d0892dee1"
format_stamp: "Formatted at 2026-09-26 20:19:48 on dist-test-slave-vnbh"
server_key: "08ffec920457d5c0f8a19f1ebfff63cd"
server_key_iv: "0b7a29de3cdfc3fce5cf2b0dc63891dd"
server_key_version: "encryptionkey@0"
I20260926 20:19:48.758018 16886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "069a933aab574a7aa008844d0892dee1"
format_stamp: "Formatted at 2026-09-26 20:19:48 on dist-test-slave-vnbh"
server_key: "08ffec920457d5c0f8a19f1ebfff63cd"
server_key_iv: "0b7a29de3cdfc3fce5cf2b0dc63891dd"
server_key_version: "encryptionkey@0"
I20260926 20:19:48.762174 16886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:19:48.764614 16901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:48.766126 16886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:19:48.766232 16886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "069a933aab574a7aa008844d0892dee1"
format_stamp: "Formatted at 2026-09-26 20:19:48 on dist-test-slave-vnbh"
server_key: "08ffec920457d5c0f8a19f1ebfff63cd"
server_key_iv: "0b7a29de3cdfc3fce5cf2b0dc63891dd"
server_key_version: "encryptionkey@0"
I20260926 20:19:48.766325 16886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:48.791977 16886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:48.795855 16886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:48.796034 16886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:48.796619 16886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:48.797596 16886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:48.797648 16886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:48.797693 16886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:48.797708 16886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:48.807663 16886 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:40551
I20260926 20:19:48.807680 17014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:40551 every 8 connection(s)
I20260926 20:19:48.808634 16886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:19:48.813221 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 16886
I20260926 20:19:48.813416 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:19:48.813777 32223 external_mini_cluster.cc:1428] Setting key 22d5c6b82e7dffead28bb53495d549e7
Sep 26 20:19:48 dist-test-slave-vnbh krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453988, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
I20260926 20:19:48.825697 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:48.810648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:60217 (local address 127.31.119.254:39053)
0926 20:19:48.810818 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:48.810823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:48.811882 (+ 1059us) server_negotiation.cc:407] Connection header received
0926 20:19:48.813352 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:48.813356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:48.813429 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:48.813537 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:48.814975 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:48.815672 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:48.817658 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:48.818119 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:48.818261 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:48.821523 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:48.821527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:48.821529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:48.821556 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:48.823188 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:48.823827 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:48.823832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:48.823841 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:48.823930 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:48.824368 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:48.824372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:48.824373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:48.824576 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:48.824733 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:48.825370 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 20:19:48.825522 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:19:48.826689 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:48.811107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:39053 (local address 127.31.119.193:60217)
0926 20:19:48.811603 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:48.811650 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:48.813090 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:48.813743 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:48.813751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:48.814175 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:48.814816 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:48.814831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:48.815853 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:48.815856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:48.817510 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:48.817528 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:48.818357 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:48.818360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:48.818515 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:48.819219 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:48.819246 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:48.821219 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:48.823366 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:48.823374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:48.823393 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:48.823699 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:48.824044 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:48.824047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:48.824049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:48.824265 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:48.824733 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:48.824742 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:48.825093 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 20:19:48.825341 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 20:19:48.825723 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
I20260926 20:19:48.828251 17015 heartbeater.cc:347] Connected to a master server at 127.31.119.254:39053
I20260926 20:19:48.828567 17015 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:48.829262 17015 heartbeater.cc:511] Master 127.31.119.254:39053 requested a full tablet report, sending...
I20260926 20:19:48.831066 16824 ts_manager.cc:194] Registered new tserver with Master: 069a933aab574a7aa008844d0892dee1 (127.31.119.193:40551)
I20260926 20:19:48.832350 16824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:60217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:19:48.845647 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:39053
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:48.960790 17020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:48.961064 17020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:48.961166 17020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:48.965322 17020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:48.965387 17020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:48.965508 17020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:19:48.971061 17020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:48.972235 17020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:48.973377 17020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:19:48.983457 17020 server_base.cc:1034] running on GCE node
W20260926 20:19:48.983419 17026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:48.983435 17025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:48.983410 17028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:48.984140 17020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:48.984859 17020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:48.986132 17020 hybrid_clock.cc:648] HybridClock initialized: now 1790453988986110 us; error 35 us; skew 500 ppm
Sep 26 20:19:48 dist-test-slave-vnbh krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453988, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:48.989118 17020 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:19:48.990145 17020 webserver.cc:460] Webserver started at http://127.31.119.194:46645/ using document root <none> and password file <none>
I20260926 20:19:48.990751 17020 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:48.990855 17020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:48.991070 17020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:48.993362 17020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "de07cd3ae3e24d61bb47930069942b04"
format_stamp: "Formatted at 2026-09-26 20:19:48 on dist-test-slave-vnbh"
server_key: "f7b61650ed1346b8ea90fb75a335609e"
server_key_iv: "861bf8ae23e9de749c1e8f45050bed14"
server_key_version: "encryptionkey@0"
I20260926 20:19:48.993954 17020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "de07cd3ae3e24d61bb47930069942b04"
format_stamp: "Formatted at 2026-09-26 20:19:48 on dist-test-slave-vnbh"
server_key: "f7b61650ed1346b8ea90fb75a335609e"
server_key_iv: "861bf8ae23e9de749c1e8f45050bed14"
server_key_version: "encryptionkey@0"
I20260926 20:19:48.998049 17020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:19:49.000684 17035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:49.002138 17020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:49.002328 17020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "de07cd3ae3e24d61bb47930069942b04"
format_stamp: "Formatted at 2026-09-26 20:19:48 on dist-test-slave-vnbh"
server_key: "f7b61650ed1346b8ea90fb75a335609e"
server_key_iv: "861bf8ae23e9de749c1e8f45050bed14"
server_key_version: "encryptionkey@0"
I20260926 20:19:49.002473 17020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:49.029395 17020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:49.033041 17020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:49.033375 17020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:49.034057 17020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:49.034976 17020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:49.035058 17020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:49.035133 17020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:49.035171 17020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:49.044425 17020 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:34701
I20260926 20:19:49.044442 17148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:34701 every 8 connection(s)
I20260926 20:19:49.045519 17020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:19:49.052994 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 17020
I20260926 20:19:49.053139 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:19:49.053500 32223 external_mini_cluster.cc:1428] Setting key dd9c3c7ac7396c92c0bad15f891f4ab4
Sep 26 20:19:49 dist-test-slave-vnbh krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453988, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:49.061194 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.047614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:44559 (local address 127.31.119.254:39053)
0926 20:19:49.047758 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:49.047761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:49.048664 (+ 903us) server_negotiation.cc:407] Connection header received
0926 20:19:49.050118 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:49.050121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:49.050178 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:49.050292 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:49.051508 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.052043 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.053582 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.053846 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.053979 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.057007 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:49.057011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:49.057013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:49.057038 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:49.058657 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:49.059298 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:49.059303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:49.059305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:49.059416 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:49.059963 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:49.059967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:49.059969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:49.060178 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:49.060318 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:49.060888 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 20:19:49.061028 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:19:49.062129 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.048059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:39053 (local address 127.31.119.194:44559)
0926 20:19:49.048510 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:49.048553 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:49.049892 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:49.050470 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:49.050477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:49.050828 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:49.051355 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.051369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.052175 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.052179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.053406 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.053419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.054030 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.054034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.054234 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.054899 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:49.054919 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:49.056794 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:49.058870 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:49.058877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:49.058894 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:49.059172 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:49.059524 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:49.059527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:49.059529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:49.059834 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:49.060317 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:49.060327 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:49.060623 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 20:19:49.060844 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 20:19:49.061220 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
I20260926 20:19:49.063521 17149 heartbeater.cc:347] Connected to a master server at 127.31.119.254:39053
I20260926 20:19:49.063807 17149 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:49.064301 17149 heartbeater.cc:511] Master 127.31.119.254:39053 requested a full tablet report, sending...
I20260926 20:19:49.065567 16824 ts_manager.cc:194] Registered new tserver with Master: de07cd3ae3e24d61bb47930069942b04 (127.31.119.194:34701)
I20260926 20:19:49.066330 16824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:44559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:19:49.085726 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:39053
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:49.202137 17154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:49.202428 17154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:49.202473 17154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:49.206737 17154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:49.206810 17154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:49.206874 17154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:19:49.212580 17154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:49.213778 17154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:49.214910 17154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:49.225432 17162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:49.225522 17154 server_base.cc:1034] running on GCE node
W20260926 20:19:49.225432 17159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:49.225450 17160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:49.226166 17154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:49.226938 17154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:49.228118 17154 hybrid_clock.cc:648] HybridClock initialized: now 1790453989228082 us; error 62 us; skew 500 ppm
Sep 26 20:19:49 dist-test-slave-vnbh krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:49.231621 17154 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:19:49.232685 17154 webserver.cc:460] Webserver started at http://127.31.119.195:38521/ using document root <none> and password file <none>
I20260926 20:19:49.233368 17154 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:49.233444 17154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:49.233669 17154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:49.235790 17154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "66060d2e2e43437a86cbb4ee9367f38c"
format_stamp: "Formatted at 2026-09-26 20:19:49 on dist-test-slave-vnbh"
server_key: "f55d44e3800cff9d4049a84f2a784008"
server_key_iv: "04f4bff9e3f51f2a488f5ef57da05582"
server_key_version: "encryptionkey@0"
I20260926 20:19:49.236382 17154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "66060d2e2e43437a86cbb4ee9367f38c"
format_stamp: "Formatted at 2026-09-26 20:19:49 on dist-test-slave-vnbh"
server_key: "f55d44e3800cff9d4049a84f2a784008"
server_key_iv: "04f4bff9e3f51f2a488f5ef57da05582"
server_key_version: "encryptionkey@0"
I20260926 20:19:49.240272 17154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:19:49.242985 17169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:49.244490 17154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:19:49.244602 17154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "66060d2e2e43437a86cbb4ee9367f38c"
format_stamp: "Formatted at 2026-09-26 20:19:49 on dist-test-slave-vnbh"
server_key: "f55d44e3800cff9d4049a84f2a784008"
server_key_iv: "04f4bff9e3f51f2a488f5ef57da05582"
server_key_version: "encryptionkey@0"
I20260926 20:19:49.244776 17154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:49.263602 17154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:49.267253 17154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:49.267491 17154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:49.268134 17154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:49.269011 17154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:49.269057 17154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:49.269098 17154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:49.269158 17154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:49.278815 17154 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:43439
I20260926 20:19:49.278839 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:43439 every 8 connection(s)
I20260926 20:19:49.279840 17154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:19:49.283193 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 17154
I20260926 20:19:49.283360 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:19:49.283664 32223 external_mini_cluster.cc:1428] Setting key df776ec9aa26d5b76a63826500526a22
Sep 26 20:19:49 dist-test-slave-vnbh krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:49.295068 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.281772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:50317 (local address 127.31.119.254:39053)
0926 20:19:49.281930 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:49.281934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:49.282923 (+ 989us) server_negotiation.cc:407] Connection header received
0926 20:19:49.284495 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:49.284500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:49.284562 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:49.284643 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:49.286321 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.286914 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.288127 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.288408 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.288514 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.291230 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:49.291234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:49.291236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:49.291264 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:49.292604 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:49.293187 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:49.293190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:49.293192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:49.293276 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:49.293974 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:49.293977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:49.293978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:49.294185 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:49.294272 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:49.294791 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 20:19:49.294924 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:19:49.295959 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.282212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:39053 (local address 127.31.119.195:50317)
0926 20:19:49.282685 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:49.282730 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:49.284232 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:49.284843 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:49.284850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:49.285414 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:49.286148 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.286169 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.287031 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.287033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.287958 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.287967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.288564 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.288567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.288714 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.289401 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:49.289418 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:49.291049 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:49.292715 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:49.292720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:49.292733 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:49.293032 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:49.293677 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:49.293680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:49.293681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:49.293873 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:49.294284 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:49.294290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:49.294553 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 20:19:49.294718 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 20:19:49.295036 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260926 20:19:49.297546 17283 heartbeater.cc:347] Connected to a master server at 127.31.119.254:39053
I20260926 20:19:49.297818 17283 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:49.298453 17283 heartbeater.cc:511] Master 127.31.119.254:39053 requested a full tablet report, sending...
I20260926 20:19:49.299543 16824 ts_manager.cc:194] Registered new tserver with Master: 66060d2e2e43437a86cbb4ee9367f38c (127.31.119.195:43439)
I20260926 20:19:49.300235 16824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:50317
I20260926 20:19:49.310869 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:19:49.320981 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.313039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.31.119.254:39053)
0926 20:19:49.313227 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:49.313232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:49.313347 (+ 115us) server_negotiation.cc:407] Connection header received
0926 20:19:49.313498 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:49.313501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:49.313547 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:49.313619 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:49.314492 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.315031 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.315865 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.316067 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.316141 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.317807 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:49.317810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:49.317812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:49.317838 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:49.319309 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:49.319708 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:49.319711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:49.319712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:49.319825 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:49.320064 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:49.320066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:49.320067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:49.320288 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:49.320396 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:49.320603 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 20:19:49.320709 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:19:49.324249 16824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:19:49.327767 16824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:19:49.343466 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.337093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:34701 (local address 127.0.0.1:43286)
0926 20:19:49.337957 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:49.338012 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:49.338216 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:49.339016 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:49.339022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:49.339045 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:49.339262 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.339274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.340827 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.340830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.341997 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.342006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.342845 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.342849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.342958 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.342995 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:19:49.343140 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 20:19:49.343287 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":72,"thread_start_us":85,"threads_started":1}
I20260926 20:19:49.343452 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.337251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.31.119.194:34701)
0926 20:19:49.337737 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:49.337753 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:49.338108 (+ 355us) server_negotiation.cc:407] Connection header received
0926 20:19:49.338432 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:49.338441 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:49.338666 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:49.338796 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:49.339403 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.340650 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.342129 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.342696 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.342861 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.342988 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:49.343182 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 20:19:49.343280 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":112,"threads_started":1}
I20260926 20:19:49.343819 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.337537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.31.119.193:40551)
0926 20:19:49.337942 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:49.337949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:49.338495 (+ 546us) server_negotiation.cc:407] Connection header received
0926 20:19:49.338722 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:49.338728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:49.338930 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:49.339107 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:49.339427 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.340215 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.341822 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.342701 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.342828 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.342970 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:49.343576 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 20:19:49.343676 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":105,"threads_started":1}
I20260926 20:19:49.344076 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.338194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.31.119.195:43439)
0926 20:19:49.338758 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:49.338770 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:49.338801 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:19:49.338870 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:49.338874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:49.339135 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:49.339322 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:49.339586 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.340690 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.342203 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.343018 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.343646 (+ 628us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.343774 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:49.343823 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:19:49.343921 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":138,"threads_started":1}
I20260926 20:19:49.343511 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.337905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:43439 (local address 127.0.0.1:37432)
0926 20:19:49.338393 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:49.338406 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:49.338490 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:49.339328 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:49.339334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:49.339357 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:49.339457 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.339465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.340839 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.340843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.342069 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.342078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.343140 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.343143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.343233 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.343258 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:19:49.343300 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 20:19:49.343348 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":43,"thread_start_us":90,"threads_started":1}
I20260926 20:19:49.343511 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.337140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40551 (local address 127.0.0.1:33738)
0926 20:19:49.337957 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:49.338012 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:49.338566 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:49.339081 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:49.339084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:49.339101 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:49.339291 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.339299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.340346 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.340352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.341678 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.341688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.343038 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.343043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.343154 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.343177 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:19:49.343255 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:19:49.343335 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":38,"thread_start_us":153,"threads_started":1}
I20260926 20:19:49.347558 17217 tablet_service.cc:1467] Processing CreateTablet for tablet 1f16ef6b7c234ed5beafa68906170be9 (DEFAULT_TABLE table=test-table [id=1cf1350fc5cf4e00acc02d65f072be2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:49.347971 16949 tablet_service.cc:1467] Processing CreateTablet for tablet 1f16ef6b7c234ed5beafa68906170be9 (DEFAULT_TABLE table=test-table [id=1cf1350fc5cf4e00acc02d65f072be2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:49.348847 17217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f16ef6b7c234ed5beafa68906170be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:49.348966 16949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f16ef6b7c234ed5beafa68906170be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:49.350270 17083 tablet_service.cc:1467] Processing CreateTablet for tablet 1f16ef6b7c234ed5beafa68906170be9 (DEFAULT_TABLE table=test-table [id=1cf1350fc5cf4e00acc02d65f072be2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:49.351135 17083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f16ef6b7c234ed5beafa68906170be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:49.356423 17300 tablet_bootstrap.cc:492] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1: Bootstrap starting.
I20260926 20:19:49.357264 17301 tablet_bootstrap.cc:492] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: Bootstrap starting.
I20260926 20:19:49.357961 17302 tablet_bootstrap.cc:492] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04: Bootstrap starting.
I20260926 20:19:49.359162 17300 tablet_bootstrap.cc:654] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:49.359426 17301 tablet_bootstrap.cc:654] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:49.359977 17300 log.cc:824] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:49.360018 17302 tablet_bootstrap.cc:654] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:49.360819 17301 log.cc:824] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:49.361227 17302 log.cc:824] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:49.362237 17300 tablet_bootstrap.cc:492] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1: No bootstrap required, opened a new log
I20260926 20:19:49.362464 17300 ts_tablet_manager.cc:1397] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:19:49.362793 17301 tablet_bootstrap.cc:492] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: No bootstrap required, opened a new log
I20260926 20:19:49.362978 17301 ts_tablet_manager.cc:1397] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 20:19:49.363091 17302 tablet_bootstrap.cc:492] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04: No bootstrap required, opened a new log
I20260926 20:19:49.363258 17302 ts_tablet_manager.cc:1397] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:19:49.366473 17300 raft_consensus.cc:348] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 } } peers { permanent_uuid: "069a933aab574a7aa008844d0892dee1" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40551 } } peers { permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 } }
I20260926 20:19:49.366747 17300 raft_consensus.cc:374] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:49.366803 17300 raft_consensus.cc:729] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069a933aab574a7aa008844d0892dee1, State: Initialized, Role: FOLLOWER
I20260926 20:19:49.366978 17302 raft_consensus.cc:348] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 } } peers { permanent_uuid: "069a933aab574a7aa008844d0892dee1" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40551 } } peers { permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 } }
I20260926 20:19:49.366978 17301 raft_consensus.cc:348] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 } } peers { permanent_uuid: "069a933aab574a7aa008844d0892dee1" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40551 } } peers { permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 } }
I20260926 20:19:49.367233 17301 raft_consensus.cc:374] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:49.367233 17302 raft_consensus.cc:374] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:49.367302 17301 raft_consensus.cc:729] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66060d2e2e43437a86cbb4ee9367f38c, State: Initialized, Role: FOLLOWER
I20260926 20:19:49.367311 17302 raft_consensus.cc:729] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de07cd3ae3e24d61bb47930069942b04, State: Initialized, Role: FOLLOWER
I20260926 20:19:49.367311 17300 consensus_queue.cc:260] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 } } peers { permanent_uuid: "069a933aab574a7aa008844d0892dee1" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40551 } } peers { permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 } }
I20260926 20:19:49.367751 17301 consensus_queue.cc:260] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 } } peers { permanent_uuid: "069a933aab574a7aa008844d0892dee1" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40551 } } peers { permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 } }
I20260926 20:19:49.367753 17302 consensus_queue.cc:260] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 } } peers { permanent_uuid: "069a933aab574a7aa008844d0892dee1" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40551 } } peers { permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 } }
I20260926 20:19:49.368155 17015 heartbeater.cc:503] Master 127.31.119.254:39053 was elected leader, sending a full tablet report...
I20260926 20:19:49.368445 17300 ts_tablet_manager.cc:1428] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:19:49.368671 17302 ts_tablet_manager.cc:1428] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 20:19:49.369028 17149 heartbeater.cc:503] Master 127.31.119.254:39053 was elected leader, sending a full tablet report...
I20260926 20:19:49.369356 17283 heartbeater.cc:503] Master 127.31.119.254:39053 was elected leader, sending a full tablet report...
I20260926 20:19:49.369668 17301 ts_tablet_manager.cc:1428] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260926 20:19:49.421793 17306 raft_consensus.cc:482] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:19:49.422045 17306 raft_consensus.cc:504] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 } } peers { permanent_uuid: "069a933aab574a7aa008844d0892dee1" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40551 } } peers { permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 } }
I20260926 20:19:49.423242 17306 leader_election.cc:288] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de07cd3ae3e24d61bb47930069942b04 (127.31.119.194:34701), 66060d2e2e43437a86cbb4ee9367f38c (127.31.119.195:43439)
I20260926 20:19:49.428566 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.423610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:38053 (local address 127.31.119.195:43439)
0926 20:19:49.423774 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:49.423779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:49.423923 (+ 144us) server_negotiation.cc:407] Connection header received
0926 20:19:49.424031 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:49.424037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:49.424097 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:49.424189 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:49.424551 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.425394 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.427119 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.427813 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.427980 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.428040 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:49.428334 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 20:19:49.428421 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:19:49.428566 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.423631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:40185 (local address 127.31.119.194:34701)
0926 20:19:49.423841 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:49.423845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:49.424381 (+ 536us) server_negotiation.cc:407] Connection header received
0926 20:19:49.424439 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:49.424441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:49.424483 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:49.424590 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:49.424845 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.425589 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.427119 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.427905 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.428027 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.428083 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:49.428334 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 20:19:49.428422 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:19:49.428601 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.423524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:34701 (local address 127.31.119.193:40185)
0926 20:19:49.424265 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:49.424280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:49.424367 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:49.424611 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:49.424613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:49.424628 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:49.424729 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.424733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.425677 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.425680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.426972 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.426983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.428014 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.428016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.428134 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.428179 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:19:49.428260 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:19:49.428389 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":422,"thread_start_us":126,"threads_started":1}
I20260926 20:19:49.428601 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.423518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:43439 (local address 127.31.119.193:38053)
0926 20:19:49.423824 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:49.423839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:49.423940 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:49.424220 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:49.424223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:49.424241 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:49.424428 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.424437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.425514 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.425518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.426972 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:49.426983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.427981 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.427987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:49.428131 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.428185 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 20:19:49.428260 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:19:49.428356 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":52,"thread_start_us":115,"threads_started":1}
I20260926 20:19:49.429495 17103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f16ef6b7c234ed5beafa68906170be9" candidate_uuid: "069a933aab574a7aa008844d0892dee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de07cd3ae3e24d61bb47930069942b04" is_pre_election: true
I20260926 20:19:49.429522 17237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f16ef6b7c234ed5beafa68906170be9" candidate_uuid: "069a933aab574a7aa008844d0892dee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66060d2e2e43437a86cbb4ee9367f38c" is_pre_election: true
I20260926 20:19:49.429827 17103 raft_consensus.cc:2454] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 069a933aab574a7aa008844d0892dee1 in term 0.
I20260926 20:19:49.429845 17237 raft_consensus.cc:2454] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 069a933aab574a7aa008844d0892dee1 in term 0.
I20260926 20:19:49.430377 16904 leader_election.cc:302] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 069a933aab574a7aa008844d0892dee1, 66060d2e2e43437a86cbb4ee9367f38c; no voters:
I20260926 20:19:49.430663 17306 raft_consensus.cc:2781] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:19:49.430749 17306 raft_consensus.cc:482] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:19:49.430791 17306 raft_consensus.cc:3037] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:49.431892 17306 raft_consensus.cc:504] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 } } peers { permanent_uuid: "069a933aab574a7aa008844d0892dee1" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40551 } } peers { permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 } }
I20260926 20:19:49.432253 17306 leader_election.cc:288] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [CANDIDATE]: Term 1 election: Requested vote from peers de07cd3ae3e24d61bb47930069942b04 (127.31.119.194:34701), 66060d2e2e43437a86cbb4ee9367f38c (127.31.119.195:43439)
I20260926 20:19:49.432647 17237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f16ef6b7c234ed5beafa68906170be9" candidate_uuid: "069a933aab574a7aa008844d0892dee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66060d2e2e43437a86cbb4ee9367f38c"
I20260926 20:19:49.432760 17237 raft_consensus.cc:3037] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:49.433517 17103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f16ef6b7c234ed5beafa68906170be9" candidate_uuid: "069a933aab574a7aa008844d0892dee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de07cd3ae3e24d61bb47930069942b04"
I20260926 20:19:49.433640 17103 raft_consensus.cc:3037] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:49.434096 17237 raft_consensus.cc:2454] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 069a933aab574a7aa008844d0892dee1 in term 1.
I20260926 20:19:49.434455 16904 leader_election.cc:302] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 069a933aab574a7aa008844d0892dee1, 66060d2e2e43437a86cbb4ee9367f38c; no voters:
I20260926 20:19:49.434653 17306 raft_consensus.cc:2781] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:49.434911 17306 raft_consensus.cc:686] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 1 LEADER]: Becoming Leader. State: Replica: 069a933aab574a7aa008844d0892dee1, State: Running, Role: LEADER
I20260926 20:19:49.434978 17103 raft_consensus.cc:2454] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 069a933aab574a7aa008844d0892dee1 in term 1.
I20260926 20:19:49.435315 17306 consensus_queue.cc:237] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 } } peers { permanent_uuid: "069a933aab574a7aa008844d0892dee1" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40551 } } peers { permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 } }
I20260926 20:19:49.438977 16822 catalog_manager.cc:5495] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 069a933aab574a7aa008844d0892dee1 (127.31.119.193). New cstate: current_term: 1 leader_uuid: "069a933aab574a7aa008844d0892dee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "069a933aab574a7aa008844d0892dee1" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 40551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 } health_report { overall_health: UNKNOWN } } }
I20260926 20:19:49.472914 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.468877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.31.119.193:40551)
0926 20:19:49.469025 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:49.469029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:49.469043 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:19:49.469124 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:49.469126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:49.469184 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:49.469318 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:19:49.469778 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.470366 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.471292 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.471506 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.471589 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.471984 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:19:49.472451 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:19:49.472567 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:49.472672 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:19:49.472729 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:19:49.479568 17103 raft_consensus.cc:1261] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04 [term 1 FOLLOWER]: Refusing update from remote peer 069a933aab574a7aa008844d0892dee1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:49.479568 17237 raft_consensus.cc:1261] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c [term 1 FOLLOWER]: Refusing update from remote peer 069a933aab574a7aa008844d0892dee1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:49.480348 17306 consensus_queue.cc:1035] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66060d2e2e43437a86cbb4ee9367f38c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:49.480561 17311 consensus_queue.cc:1035] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de07cd3ae3e24d61bb47930069942b04" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:49.488472 17314 mvcc.cc:205] Tried to move back new op lower bound from 7333699540900253696 to 7333699540730118144. Current Snapshot: MvccSnapshot[applied={T|T < 7333699540900253696}]
I20260926 20:19:49.490258 17316 mvcc.cc:205] Tried to move back new op lower bound from 7333699540900253696 to 7333699540730118144. Current Snapshot: MvccSnapshot[applied={T|T < 7333699540900253696}]
I20260926 20:19:49.497772 16823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:49.497956 16823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:49.500829 16823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48035303216f40e19c18dd05ab1cbf0c: Sending DeleteTablet for 3 replicas of tablet 1f16ef6b7c234ed5beafa68906170be9
I20260926 20:19:49.501793 17083 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f16ef6b7c234ed5beafa68906170be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:49 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:43286
I20260926 20:19:49.502063 17217 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f16ef6b7c234ed5beafa68906170be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:49 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:37432
I20260926 20:19:49.502228 16949 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f16ef6b7c234ed5beafa68906170be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:49 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:33738
I20260926 20:19:49.502770 17326 tablet_replica.cc:330] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1: stopping tablet replica
I20260926 20:19:49.503360 17326 raft_consensus.cc:2229] T 1f16ef6b7c234ed5beafa68906170be9 P 069a933aab574a7aa008844d0892dee1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:19:49.503600 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 16886
I20260926 20:19:49.505187 17325 tablet_replica.cc:330] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: stopping tablet replica
I20260926 20:19:49.505645 17325 raft_consensus.cc:2229] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:49.506057 17325 raft_consensus.cc:2258] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:49.507864 17324 tablet_replica.cc:330] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04: stopping tablet replica
I20260926 20:19:49.508219 17324 raft_consensus.cc:2229] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:49.508570 17324 raft_consensus.cc:2258] T 1f16ef6b7c234ed5beafa68906170be9 P de07cd3ae3e24d61bb47930069942b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:49.510471 17325 ts_tablet_manager.cc:1905] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:19:49.510804 16810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:40551 (error 108)
I20260926 20:19:49.511979 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.511615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:40551 (local address 127.0.0.1:33758)
0926 20:19:49.511781 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:49.511867 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:40551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 20:19:49.512092 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 17020
W20260926 20:19:49.512339 16810 catalog_manager.cc:4584] TS 069a933aab574a7aa008844d0892dee1 (127.31.119.193:40551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f16ef6b7c234ed5beafa68906170be9: Network error: Client connection negotiation failed: client connection to 127.31.119.193:40551: connect: Connection refused (error 111)
I20260926 20:19:49.513499 17325 ts_tablet_manager.cc:1918] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:49.513607 17325 log.cc:1196] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/1f16ef6b7c234ed5beafa68906170be9
I20260926 20:19:49.513934 17325 ts_tablet_manager.cc:1939] T 1f16ef6b7c234ed5beafa68906170be9 P 66060d2e2e43437a86cbb4ee9367f38c: Deleting consensus metadata
I20260926 20:19:49.515317 16810 catalog_manager.cc:4841] TS 66060d2e2e43437a86cbb4ee9367f38c (127.31.119.195:43439): tablet 1f16ef6b7c234ed5beafa68906170be9 (table test-table [id=1cf1350fc5cf4e00acc02d65f072be2e]) successfully deleted
I20260926 20:19:49.519853 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 17154
I20260926 20:19:49.521508 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.520171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:34701 (local address 127.0.0.1:43294)
0926 20:19:49.521265 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:49.521384 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.194:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":972}
I20260926 20:19:49.526132 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 16792
2026-09-26T20:19:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2454 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T20:19:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:19:49Z Disabled control of system clock
I20260926 20:19:49.558204 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:42087
--webserver_interface=127.31.119.254
--webserver_port=0
--builtin_ntp_servers=127.31.119.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:42087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:19:49.677608 17331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:49.677933 17331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:49.677973 17331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:49.682097 17331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:19:49.682170 17331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:49.682185 17331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:49.682202 17331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:19:49.682217 17331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:19:49.687382 17331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:42087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:42087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:49.688515 17331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:49.689949 17331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:19:49.699141 17331 server_base.cc:1034] running on GCE node
W20260926 20:19:49.699167 17337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:49.699077 17336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:49.699075 17339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:49.700013 17331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:49.701154 17331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:49.702350 17331 hybrid_clock.cc:648] HybridClock initialized: now 1790453989702327 us; error 36 us; skew 500 ppm
I20260926 20:19:49.704550 17331 webserver.cc:460] Webserver started at http://127.31.119.254:36409/ using document root <none> and password file <none>
I20260926 20:19:49.705247 17331 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:49.705391 17331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:49.705634 17331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:49.707779 17331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "8b6e012c0d6e4cd1970ee099050bcda5"
format_stamp: "Formatted at 2026-09-26 20:19:49 on dist-test-slave-vnbh"
server_key: "341920cd84345da650f00f59e980c559"
server_key_iv: "b121f437e246410e1db11a2e4fa3ce2e"
server_key_version: "encryptionkey@0"
I20260926 20:19:49.708389 17331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "8b6e012c0d6e4cd1970ee099050bcda5"
format_stamp: "Formatted at 2026-09-26 20:19:49 on dist-test-slave-vnbh"
server_key: "341920cd84345da650f00f59e980c559"
server_key_iv: "b121f437e246410e1db11a2e4fa3ce2e"
server_key_version: "encryptionkey@0"
I20260926 20:19:49.712332 17331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:19:49.715417 17345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:49.716758 17331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:19:49.716966 17331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "8b6e012c0d6e4cd1970ee099050bcda5"
format_stamp: "Formatted at 2026-09-26 20:19:49 on dist-test-slave-vnbh"
server_key: "341920cd84345da650f00f59e980c559"
server_key_iv: "b121f437e246410e1db11a2e4fa3ce2e"
server_key_version: "encryptionkey@0"
I20260926 20:19:49.717097 17331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:49.736363 17331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:49.737154 17331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:49.737421 17331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:49.744801 17331 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:42087
I20260926 20:19:49.744807 17397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:42087 every 8 connection(s)
I20260926 20:19:49.745874 17331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:19:49.749202 17398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:49.754987 17398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5: Bootstrap starting.
I20260926 20:19:49.755388 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 17331
I20260926 20:19:49.755517 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:19:49.755820 32223 external_mini_cluster.cc:1428] Setting key 1e330ae7ae1e778c7ada2573c3aaef73
I20260926 20:19:49.757936 17398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:49.758723 17398 log.cc:824] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:49.760732 17398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5: No bootstrap required, opened a new log
I20260926 20:19:49.764564 17398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6e012c0d6e4cd1970ee099050bcda5" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42087 } }
I20260926 20:19:49.764886 17398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:49.764948 17398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6e012c0d6e4cd1970ee099050bcda5, State: Initialized, Role: FOLLOWER
I20260926 20:19:49.765484 17398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6e012c0d6e4cd1970ee099050bcda5" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42087 } }
I20260926 20:19:49.765653 17398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:19:49.765758 17398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:19:49.765893 17398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:49.766489 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:49.757374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.31.119.254:42087)
0926 20:19:49.758128 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:49.758147 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:49.758184 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:19:49.758905 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:49.758925 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:49.759442 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:49.759836 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:49.760273 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.761420 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.762363 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:49.762862 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:49.763044 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:49.764085 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:49.764090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:49.764104 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:49.764511 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:49.764582 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:49.764603 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:49.764774 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:49.764960 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 20:19:49.765440 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":142,"threads_started":1}
I20260926 20:19:49.767066 17398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6e012c0d6e4cd1970ee099050bcda5" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42087 } }
I20260926 20:19:49.767418 17398 leader_election.cc:302] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b6e012c0d6e4cd1970ee099050bcda5; no voters:
I20260926 20:19:49.767779 17398 leader_election.cc:288] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:19:49.767956 17403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:49.768186 17403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [term 1 LEADER]: Becoming Leader. State: Replica: 8b6e012c0d6e4cd1970ee099050bcda5, State: Running, Role: LEADER
I20260926 20:19:49.768493 17403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6e012c0d6e4cd1970ee099050bcda5" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42087 } }
I20260926 20:19:49.769133 17398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:19:49.770330 17404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b6e012c0d6e4cd1970ee099050bcda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6e012c0d6e4cd1970ee099050bcda5" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42087 } } }
I20260926 20:19:49.770330 17405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b6e012c0d6e4cd1970ee099050bcda5. Latest consensus state: current_term: 1 leader_uuid: "8b6e012c0d6e4cd1970ee099050bcda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6e012c0d6e4cd1970ee099050bcda5" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42087 } } }
I20260926 20:19:49.770486 17405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:49.770486 17404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:49.771011 17412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:19:49.773880 17412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:19:49.779599 17412 catalog_manager.cc:1347] Generated new cluster ID: ee2b9d23175244e699e5699b9ed7bfd2
I20260926 20:19:49.779692 17412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:19:49.803095 17412 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:19:49.804071 17412 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:19:49.812029 17412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5: Generated new TSK 0
I20260926 20:19:49.812841 17412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:19:49.820482 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42087
--builtin_ntp_servers=127.31.119.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:19:49.937016 17422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:49.937369 17422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:49.937474 17422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:49.941538 17422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:49.941641 17422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:49.941740 17422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:19:49.946837 17422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:49.947996 17422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:49.949469 17422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:49.959903 17428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:49.959961 17430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:49.960083 17422 server_base.cc:1034] running on GCE node
W20260926 20:19:49.959960 17427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:49.960804 17422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:49.961611 17422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:49.962813 17422 hybrid_clock.cc:648] HybridClock initialized: now 1790453989962775 us; error 51 us; skew 500 ppm
I20260926 20:19:49.964910 17422 webserver.cc:460] Webserver started at http://127.31.119.193:40447/ using document root <none> and password file <none>
I20260926 20:19:49.965600 17422 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:49.965703 17422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:49.965983 17422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:49.968175 17422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "ec229d6ccfbd4b4ea00f387da9138a6b"
format_stamp: "Formatted at 2026-09-26 20:19:49 on dist-test-slave-vnbh"
server_key: "9ae6403b087322c5924469aeed746ddd"
server_key_iv: "6c1dbb8e46c2de7564dd9325e5c7e0b8"
server_key_version: "encryptionkey@0"
I20260926 20:19:49.968823 17422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "ec229d6ccfbd4b4ea00f387da9138a6b"
format_stamp: "Formatted at 2026-09-26 20:19:49 on dist-test-slave-vnbh"
server_key: "9ae6403b087322c5924469aeed746ddd"
server_key_iv: "6c1dbb8e46c2de7564dd9325e5c7e0b8"
server_key_version: "encryptionkey@0"
I20260926 20:19:49.973187 17422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 20:19:49.975808 17436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:49.977448 17422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:49.977602 17422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "ec229d6ccfbd4b4ea00f387da9138a6b"
format_stamp: "Formatted at 2026-09-26 20:19:49 on dist-test-slave-vnbh"
server_key: "9ae6403b087322c5924469aeed746ddd"
server_key_iv: "6c1dbb8e46c2de7564dd9325e5c7e0b8"
server_key_version: "encryptionkey@0"
I20260926 20:19:49.977746 17422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:50.007929 17422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:50.008754 17422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:50.008982 17422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:50.009665 17422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:50.010731 17422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:50.010819 17422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:50.010890 17422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:50.010941 17422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:50.020987 17422 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:36839
I20260926 20:19:50.021039 17549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:36839 every 8 connection(s)
I20260926 20:19:50.022117 17422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:19:50.028070 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 17422
I20260926 20:19:50.028209 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:19:50.028546 32223 external_mini_cluster.cc:1428] Setting key b0cc6a11225908efb86e4384c75e47f7
I20260926 20:19:50.030963 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42087
--builtin_ntp_servers=127.31.119.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:19:50.033962 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.024163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:54143 (local address 127.31.119.254:42087)
0926 20:19:50.024358 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:50.024364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:50.025643 (+ 1279us) server_negotiation.cc:407] Connection header received
0926 20:19:50.027094 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:50.027097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:50.027150 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:50.027240 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:50.028515 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.029465 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.031333 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.031581 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.031680 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.033377 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:50.033381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:50.033384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:50.033409 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:50.033433 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:50.033442 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:50.033544 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:50.033731 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 20:19:50.033806 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 20:19:50.035156 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.024639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:42087 (local address 127.31.119.193:54143)
0926 20:19:50.025477 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:50.025523 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:50.026842 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:50.027424 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:50.027432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:50.027524 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:50.028320 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.028344 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.029599 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.029602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.031023 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.031038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.031786 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.031793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.032069 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.032978 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:19:50.032980 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:50.033035 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:19:50.033117 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:50.033549 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:50.033562 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:50.033569 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:19:50.033737 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 20:19:50.034099 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":117,"threads_started":1}
I20260926 20:19:50.036847 17550 heartbeater.cc:347] Connected to a master server at 127.31.119.254:42087
I20260926 20:19:50.037149 17550 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:50.037838 17550 heartbeater.cc:511] Master 127.31.119.254:42087 requested a full tablet report, sending...
I20260926 20:19:50.039855 17361 ts_manager.cc:194] Registered new tserver with Master: ec229d6ccfbd4b4ea00f387da9138a6b (127.31.119.193:36839)
I20260926 20:19:50.041730 17361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.193:54143
W20260926 20:19:50.153355 17553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:50.153692 17553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:50.153739 17553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:50.158826 17553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:50.158916 17553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:50.158984 17553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:19:50.164746 17553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:50.166183 17553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:50.167749 17553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:50.179513 17561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:50.179513 17558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:50.179534 17559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:50.179795 17553 server_base.cc:1034] running on GCE node
I20260926 20:19:50.180325 17553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:50.181085 17553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:50.182298 17553 hybrid_clock.cc:648] HybridClock initialized: now 1790453990182276 us; error 43 us; skew 500 ppm
I20260926 20:19:50.184476 17553 webserver.cc:460] Webserver started at http://127.31.119.194:33485/ using document root <none> and password file <none>
I20260926 20:19:50.185102 17553 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:50.185168 17553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:50.185426 17553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:50.187600 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "e1486b2ecf4a4ad19c91b9b61946c700"
format_stamp: "Formatted at 2026-09-26 20:19:50 on dist-test-slave-vnbh"
server_key: "235fcfa0b7afa3f0302e9690da8714ed"
server_key_iv: "26660267fadf73630aa78f4baff9f743"
server_key_version: "encryptionkey@0"
I20260926 20:19:50.188156 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "e1486b2ecf4a4ad19c91b9b61946c700"
format_stamp: "Formatted at 2026-09-26 20:19:50 on dist-test-slave-vnbh"
server_key: "235fcfa0b7afa3f0302e9690da8714ed"
server_key_iv: "26660267fadf73630aa78f4baff9f743"
server_key_version: "encryptionkey@0"
I20260926 20:19:50.192201 17553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:19:50.195017 17567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:50.196602 17553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:19:50.196725 17553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "e1486b2ecf4a4ad19c91b9b61946c700"
format_stamp: "Formatted at 2026-09-26 20:19:50 on dist-test-slave-vnbh"
server_key: "235fcfa0b7afa3f0302e9690da8714ed"
server_key_iv: "26660267fadf73630aa78f4baff9f743"
server_key_version: "encryptionkey@0"
I20260926 20:19:50.196826 17553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:50.208163 17553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:50.208892 17553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:50.209065 17553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:50.209750 17553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:50.210749 17553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:50.210799 17553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:50.210852 17553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:50.210870 17553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:50.221208 17553 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:43341
I20260926 20:19:50.221257 17680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:43341 every 8 connection(s)
I20260926 20:19:50.222420 17553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:19:50.228578 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 17553
I20260926 20:19:50.228708 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:19:50.229063 32223 external_mini_cluster.cc:1428] Setting key 0975e58a9d8589da1a04bcbaf0ad3ec7
I20260926 20:19:50.231545 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42087
--builtin_ntp_servers=127.31.119.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:19:50.233727 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.224480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:41751 (local address 127.31.119.254:42087)
0926 20:19:50.224639 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:50.224644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:50.225948 (+ 1304us) server_negotiation.cc:407] Connection header received
0926 20:19:50.227571 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:50.227574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:50.227633 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:50.227726 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:50.228905 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.229654 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.231284 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.231547 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.231690 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.233091 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:50.233095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:50.233098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:50.233114 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:50.233148 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:50.233158 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:50.233256 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:50.233496 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 20:19:50.233569 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:19:50.234747 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.224978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:42087 (local address 127.31.119.194:41751)
0926 20:19:50.225775 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:50.225823 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:50.227287 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:50.227945 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:50.227953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:50.228054 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:50.228619 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.228632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.229819 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.229823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.231125 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.231136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.231741 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.231746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.231988 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.232758 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:19:50.232760 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:50.232811 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:19:50.232898 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:50.233275 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:50.233320 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:50.233330 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:19:50.233495 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 20:19:50.233860 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":116,"threads_started":1}
I20260926 20:19:50.236323 17681 heartbeater.cc:347] Connected to a master server at 127.31.119.254:42087
I20260926 20:19:50.236622 17681 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:50.237423 17681 heartbeater.cc:511] Master 127.31.119.254:42087 requested a full tablet report, sending...
I20260926 20:19:50.238737 17361 ts_manager.cc:194] Registered new tserver with Master: e1486b2ecf4a4ad19c91b9b61946c700 (127.31.119.194:43341)
I20260926 20:19:50.239413 17361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.194:41751
W20260926 20:19:50.351035 17684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:50.351349 17684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:50.351420 17684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:50.355547 17684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:50.355634 17684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:50.355726 17684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:19:50.360651 17684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:50.361864 17684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:50.363327 17684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:50.373598 17690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:50.373607 17692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:50.373803 17684 server_base.cc:1034] running on GCE node
W20260926 20:19:50.373607 17689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:50.374420 17684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:50.375192 17684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:50.376360 17684 hybrid_clock.cc:648] HybridClock initialized: now 1790453990376341 us; error 34 us; skew 500 ppm
I20260926 20:19:50.378603 17684 webserver.cc:460] Webserver started at http://127.31.119.195:33907/ using document root <none> and password file <none>
I20260926 20:19:50.379252 17684 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:50.379317 17684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:50.379540 17684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:50.381879 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "543630ad31a941ca9962ede24e601aa3"
format_stamp: "Formatted at 2026-09-26 20:19:50 on dist-test-slave-vnbh"
server_key: "115f7611a73eb0d5d0fa1229435e075e"
server_key_iv: "e7a967f949fddf94b6b789740b2e5e30"
server_key_version: "encryptionkey@0"
I20260926 20:19:50.382449 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "543630ad31a941ca9962ede24e601aa3"
format_stamp: "Formatted at 2026-09-26 20:19:50 on dist-test-slave-vnbh"
server_key: "115f7611a73eb0d5d0fa1229435e075e"
server_key_iv: "e7a967f949fddf94b6b789740b2e5e30"
server_key_version: "encryptionkey@0"
I20260926 20:19:50.386369 17684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:19:50.388892 17698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:50.390280 17684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:50.390429 17684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "543630ad31a941ca9962ede24e601aa3"
format_stamp: "Formatted at 2026-09-26 20:19:50 on dist-test-slave-vnbh"
server_key: "115f7611a73eb0d5d0fa1229435e075e"
server_key_iv: "e7a967f949fddf94b6b789740b2e5e30"
server_key_version: "encryptionkey@0"
I20260926 20:19:50.390542 17684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:50.416638 17684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:50.417569 17684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:50.417743 17684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:50.418313 17684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:50.419322 17684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:50.419402 17684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:50.419473 17684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:50.419526 17684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:50.428828 17684 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:43865
I20260926 20:19:50.428903 17811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:43865 every 8 connection(s)
I20260926 20:19:50.429863 17684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:19:50.438925 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 17684
I20260926 20:19:50.439082 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:19:50.439414 32223 external_mini_cluster.cc:1428] Setting key 3b755c3b8d149afffad0380369742d74
I20260926 20:19:50.439457 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.431962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:58585 (local address 127.31.119.254:42087)
0926 20:19:50.432122 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:50.432125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:50.433191 (+ 1066us) server_negotiation.cc:407] Connection header received
0926 20:19:50.434631 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:50.434634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:50.434682 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:50.434813 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:50.435695 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.436284 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.437497 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.437722 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.437828 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.438920 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:50.438924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:50.438927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:50.438946 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:50.438966 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:50.438974 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:50.439057 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:50.439244 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 20:19:50.439311 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:19:50.440595 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.432366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:42087 (local address 127.31.119.195:58585)
0926 20:19:50.433029 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:50.433071 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:50.434426 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:50.434977 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:50.434983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:50.435067 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:50.435568 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.435580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.436409 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.436411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.437381 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.437390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.437865 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.437867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.438009 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.438616 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:19:50.438618 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:50.438677 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:19:50.438744 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:50.439070 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:50.439082 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:50.439088 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:19:50.439243 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 20:19:50.439558 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260926 20:19:50.441859 17812 heartbeater.cc:347] Connected to a master server at 127.31.119.254:42087
I20260926 20:19:50.442114 17812 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:50.442660 17812 heartbeater.cc:511] Master 127.31.119.254:42087 requested a full tablet report, sending...
I20260926 20:19:50.443760 17361 ts_manager.cc:194] Registered new tserver with Master: 543630ad31a941ca9962ede24e601aa3 (127.31.119.195:43865)
I20260926 20:19:50.444428 17361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.195:58585
I20260926 20:19:50.454438 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:19:50.460656 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.456758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.31.119.254:42087)
0926 20:19:50.456903 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:50.456907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:50.457140 (+ 233us) server_negotiation.cc:407] Connection header received
0926 20:19:50.457251 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:50.457254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:50.457360 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:50.457438 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:50.457705 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.458299 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.459170 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.459403 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.459495 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.460278 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:50.460281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:50.460283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:50.460294 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:50.460311 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:50.460318 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:50.460376 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:50.460486 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 20:19:50.460540 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:19:50.464095 17361 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:19:50.467314 17361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:19:50.483562 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.477942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.31.119.193:36839)
0926 20:19:50.478556 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:50.478562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:50.478582 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:19:50.478648 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:50.478653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:50.478816 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:50.479075 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:50.479333 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.480556 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.482213 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.482875 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.483024 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.483235 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:50.483310 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:19:50.483407 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":90,"threads_started":1}
I20260926 20:19:50.483558 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.477313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:36839 (local address 127.0.0.1:33000)
0926 20:19:50.477925 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:50.477938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:50.478024 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:50.478953 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:50.478956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:50.478969 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:50.479189 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.479209 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.480977 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.480981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.482079 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.482087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.483011 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.483014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.483121 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.483158 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:19:50.483261 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:19:50.483391 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":151,"thread_start_us":123,"threads_started":1}
I20260926 20:19:50.483749 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.477630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:43865 (local address 127.0.0.1:57776)
0926 20:19:50.478214 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:50.478227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:50.478328 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:50.478838 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:50.478844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:50.478889 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:50.479366 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.479373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.480725 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.480729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.482217 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.482227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.483373 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.483376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.483457 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.483483 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:19:50.483529 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 20:19:50.483598 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":290,"spinlock_wait_cycles":9984,"thread_start_us":105,"threads_started":1}
I20260926 20:19:50.484678 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.477760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.31.119.195:43865)
0926 20:19:50.478197 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:50.478209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:50.478295 (+ 86us) server_negotiation.cc:407] Connection header received
0926 20:19:50.478434 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:50.478441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:50.478650 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:50.479042 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:50.479471 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.480598 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.482417 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.483211 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.484092 (+ 881us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.484283 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:50.484350 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 20:19:50.484449 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":96,"threads_started":1}
I20260926 20:19:50.485074 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.476818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:43341 (local address 127.0.0.1:46074)
0926 20:19:50.477169 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:50.477229 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:50.477384 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:50.478842 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:50.478845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:50.478882 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:50.479189 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.479199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.482050 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.482054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.483270 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.483283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.484648 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.484679 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.484776 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.484816 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:19:50.484870 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:19:50.484944 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":104,"spinlock_wait_cycles":1408,"thread_start_us":105,"threads_started":1}
I20260926 20:19:50.485772 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.477009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.31.119.194:43341)
0926 20:19:50.478323 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:50.478337 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:50.478363 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:19:50.478445 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:50.478452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:50.478642 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:50.478822 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:50.480723 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.481917 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.483408 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.484506 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.485262 (+ 756us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.485473 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:50.485526 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 20:19:50.485615 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":126,"threads_started":1}
I20260926 20:19:50.486951 17484 tablet_service.cc:1467] Processing CreateTablet for tablet b80efd9c02dc43bb9bc2e8598a137ba7 (DEFAULT_TABLE table=test-table [id=cdfcc5710b0e4d7cbf12636ed2d597ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:50.487073 17746 tablet_service.cc:1467] Processing CreateTablet for tablet b80efd9c02dc43bb9bc2e8598a137ba7 (DEFAULT_TABLE table=test-table [id=cdfcc5710b0e4d7cbf12636ed2d597ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:50.487984 17484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80efd9c02dc43bb9bc2e8598a137ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:50.488004 17746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80efd9c02dc43bb9bc2e8598a137ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:50.490021 17615 tablet_service.cc:1467] Processing CreateTablet for tablet b80efd9c02dc43bb9bc2e8598a137ba7 (DEFAULT_TABLE table=test-table [id=cdfcc5710b0e4d7cbf12636ed2d597ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:50.490978 17615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80efd9c02dc43bb9bc2e8598a137ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:50.494815 17829 tablet_bootstrap.cc:492] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3: Bootstrap starting.
I20260926 20:19:50.497475 17829 tablet_bootstrap.cc:654] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:50.498035 17830 tablet_bootstrap.cc:492] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: Bootstrap starting.
I20260926 20:19:50.498399 17829 log.cc:824] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:50.500308 17832 tablet_bootstrap.cc:492] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b: Bootstrap starting.
I20260926 20:19:50.500598 17830 tablet_bootstrap.cc:654] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:50.500938 17829 tablet_bootstrap.cc:492] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3: No bootstrap required, opened a new log
I20260926 20:19:50.501164 17829 ts_tablet_manager.cc:1397] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:19:50.501710 17830 log.cc:824] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:50.502792 17832 tablet_bootstrap.cc:654] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:50.503646 17830 tablet_bootstrap.cc:492] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: No bootstrap required, opened a new log
I20260926 20:19:50.503654 17832 log.cc:824] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:50.503866 17830 ts_tablet_manager.cc:1397] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:19:50.505892 17832 tablet_bootstrap.cc:492] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b: No bootstrap required, opened a new log
I20260926 20:19:50.505748 17829 raft_consensus.cc:348] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 } } peers { permanent_uuid: "543630ad31a941ca9962ede24e601aa3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43865 } } peers { permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 } }
I20260926 20:19:50.506052 17829 raft_consensus.cc:374] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:50.506103 17832 ts_tablet_manager.cc:1397] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:19:50.506119 17829 raft_consensus.cc:729] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543630ad31a941ca9962ede24e601aa3, State: Initialized, Role: FOLLOWER
I20260926 20:19:50.506670 17829 consensus_queue.cc:260] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 } } peers { permanent_uuid: "543630ad31a941ca9962ede24e601aa3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43865 } } peers { permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 } }
I20260926 20:19:50.507735 17812 heartbeater.cc:503] Master 127.31.119.254:42087 was elected leader, sending a full tablet report...
I20260926 20:19:50.507822 17829 ts_tablet_manager.cc:1428] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:19:50.508399 17830 raft_consensus.cc:348] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 } } peers { permanent_uuid: "543630ad31a941ca9962ede24e601aa3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43865 } } peers { permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 } }
I20260926 20:19:50.509116 17830 raft_consensus.cc:374] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:50.509204 17830 raft_consensus.cc:729] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1486b2ecf4a4ad19c91b9b61946c700, State: Initialized, Role: FOLLOWER
I20260926 20:19:50.509717 17830 consensus_queue.cc:260] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 } } peers { permanent_uuid: "543630ad31a941ca9962ede24e601aa3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43865 } } peers { permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 } }
I20260926 20:19:50.510489 17681 heartbeater.cc:503] Master 127.31.119.254:42087 was elected leader, sending a full tablet report...
I20260926 20:19:50.510927 17830 ts_tablet_manager.cc:1428] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:19:50.511423 17832 raft_consensus.cc:348] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 } } peers { permanent_uuid: "543630ad31a941ca9962ede24e601aa3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43865 } } peers { permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 } }
I20260926 20:19:50.511682 17832 raft_consensus.cc:374] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:50.511719 17832 raft_consensus.cc:729] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec229d6ccfbd4b4ea00f387da9138a6b, State: Initialized, Role: FOLLOWER
I20260926 20:19:50.512181 17832 consensus_queue.cc:260] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 } } peers { permanent_uuid: "543630ad31a941ca9962ede24e601aa3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43865 } } peers { permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 } }
I20260926 20:19:50.513381 17832 ts_tablet_manager.cc:1428] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:19:50.513440 17550 heartbeater.cc:503] Master 127.31.119.254:42087 was elected leader, sending a full tablet report...
W20260926 20:19:50.523368 17551 tablet.cc:2369] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:19:50.680931 17813 tablet.cc:2369] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:19:50.682186 17835 raft_consensus.cc:482] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:19:50.682439 17835 raft_consensus.cc:504] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 } } peers { permanent_uuid: "543630ad31a941ca9962ede24e601aa3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43865 } } peers { permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 } }
I20260926 20:19:50.683760 17835 leader_election.cc:288] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1486b2ecf4a4ad19c91b9b61946c700 (127.31.119.194:43341), ec229d6ccfbd4b4ea00f387da9138a6b (127.31.119.193:36839)
I20260926 20:19:50.687937 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.684085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:57239 (local address 127.31.119.194:43341)
0926 20:19:50.684226 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:50.684232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:50.684429 (+ 197us) server_negotiation.cc:407] Connection header received
0926 20:19:50.684602 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:50.684605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:50.684666 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:50.684773 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:50.685160 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.685747 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.686981 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.687366 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.687501 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.687550 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:50.687735 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 20:19:50.687813 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:19:50.687937 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.683979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:43341 (local address 127.31.119.195:57239)
0926 20:19:50.684324 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:50.684364 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:50.684478 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:50.684786 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:50.684789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:50.684809 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:50.684956 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.684968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.685887 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.685890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.686841 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.686851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.687495 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.687498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.687596 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.687619 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:19:50.687719 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:19:50.687798 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":42}
I20260926 20:19:50.688082 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.684073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:36839 (local address 127.31.119.195:47635)
0926 20:19:50.684538 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:50.684553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:50.684645 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:50.684919 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:50.684921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:50.684938 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:50.685083 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.685089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.685923 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.685928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.686911 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:50.686921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.687589 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.687592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:50.687695 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.687716 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:19:50.687820 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:19:50.687886 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":201,"threads_started":1}
I20260926 20:19:50.688179 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.684191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:47635 (local address 127.31.119.193:36839)
0926 20:19:50.684361 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:50.684367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:50.684682 (+ 315us) server_negotiation.cc:407] Connection header received
0926 20:19:50.684768 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:50.684771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:50.684819 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:50.684905 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:50.685233 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.685797 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.687050 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.687452 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.687588 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.687650 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:50.687946 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 20:19:50.688054 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:19:50.688736 17635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80efd9c02dc43bb9bc2e8598a137ba7" candidate_uuid: "543630ad31a941ca9962ede24e601aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" is_pre_election: true
I20260926 20:19:50.688890 17504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80efd9c02dc43bb9bc2e8598a137ba7" candidate_uuid: "543630ad31a941ca9962ede24e601aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" is_pre_election: true
I20260926 20:19:50.689090 17635 raft_consensus.cc:2454] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 543630ad31a941ca9962ede24e601aa3 in term 0.
I20260926 20:19:50.689232 17504 raft_consensus.cc:2454] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 543630ad31a941ca9962ede24e601aa3 in term 0.
I20260926 20:19:50.689668 17700 leader_election.cc:302] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 543630ad31a941ca9962ede24e601aa3, e1486b2ecf4a4ad19c91b9b61946c700; no voters:
I20260926 20:19:50.689988 17835 raft_consensus.cc:2781] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:19:50.690066 17835 raft_consensus.cc:482] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:19:50.690099 17835 raft_consensus.cc:3037] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:50.691169 17835 raft_consensus.cc:504] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 } } peers { permanent_uuid: "543630ad31a941ca9962ede24e601aa3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43865 } } peers { permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 } }
I20260926 20:19:50.691587 17835 leader_election.cc:288] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [CANDIDATE]: Term 1 election: Requested vote from peers e1486b2ecf4a4ad19c91b9b61946c700 (127.31.119.194:43341), ec229d6ccfbd4b4ea00f387da9138a6b (127.31.119.193:36839)
I20260926 20:19:50.691951 17504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80efd9c02dc43bb9bc2e8598a137ba7" candidate_uuid: "543630ad31a941ca9962ede24e601aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b"
I20260926 20:19:50.692101 17504 raft_consensus.cc:3037] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:50.692070 17635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80efd9c02dc43bb9bc2e8598a137ba7" candidate_uuid: "543630ad31a941ca9962ede24e601aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1486b2ecf4a4ad19c91b9b61946c700"
I20260926 20:19:50.692262 17635 raft_consensus.cc:3037] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:50.693256 17504 raft_consensus.cc:2454] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543630ad31a941ca9962ede24e601aa3 in term 1.
I20260926 20:19:50.693274 17635 raft_consensus.cc:2454] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543630ad31a941ca9962ede24e601aa3 in term 1.
I20260926 20:19:50.693617 17700 leader_election.cc:302] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 543630ad31a941ca9962ede24e601aa3, e1486b2ecf4a4ad19c91b9b61946c700; no voters:
I20260926 20:19:50.693843 17835 raft_consensus.cc:2781] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:50.694082 17835 raft_consensus.cc:686] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 1 LEADER]: Becoming Leader. State: Replica: 543630ad31a941ca9962ede24e601aa3, State: Running, Role: LEADER
I20260926 20:19:50.694482 17835 consensus_queue.cc:237] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 } } peers { permanent_uuid: "543630ad31a941ca9962ede24e601aa3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43865 } } peers { permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 } }
I20260926 20:19:50.697849 17362 catalog_manager.cc:5495] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 543630ad31a941ca9962ede24e601aa3 (127.31.119.195). New cstate: current_term: 1 leader_uuid: "543630ad31a941ca9962ede24e601aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "543630ad31a941ca9962ede24e601aa3" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 } health_report { overall_health: UNKNOWN } } }
W20260926 20:19:50.723730 17682 tablet.cc:2369] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:19:50.724982 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.721014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.31.119.195:43865)
0926 20:19:50.721159 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:50.721163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:50.721179 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:19:50.721370 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:50.721373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:50.721434 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:50.721560 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:19:50.721880 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.722420 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.723302 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:50.723590 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:50.723694 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:50.724090 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:19:50.724561 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:19:50.724735 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:50.724776 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 20:19:50.724840 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:19:50.731257 17635 raft_consensus.cc:1261] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700 [term 1 FOLLOWER]: Refusing update from remote peer 543630ad31a941ca9962ede24e601aa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:50.731281 17504 raft_consensus.cc:1261] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b [term 1 FOLLOWER]: Refusing update from remote peer 543630ad31a941ca9962ede24e601aa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:50.732107 17839 consensus_queue.cc:1035] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1486b2ecf4a4ad19c91b9b61946c700" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 43341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:50.732326 17835 consensus_queue.cc:1035] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec229d6ccfbd4b4ea00f387da9138a6b" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 36839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:50.739321 17842 mvcc.cc:205] Tried to move back new op lower bound from 7333699546026516480 to 7333699545887387648. Current Snapshot: MvccSnapshot[applied={T|T < 7333699546026516480}]
I20260926 20:19:50.739845 17844 mvcc.cc:205] Tried to move back new op lower bound from 7333699546026516480 to 7333699545887387648. Current Snapshot: MvccSnapshot[applied={T|T < 7333699546026516480}]
I20260926 20:19:50.742511 17843 mvcc.cc:205] Tried to move back new op lower bound from 7333699546026516480 to 7333699545887387648. Current Snapshot: MvccSnapshot[applied={T|T < 7333699546026516480}]
I20260926 20:19:50.749136 17362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:50.749372 17362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:50.751880 17362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b6e012c0d6e4cd1970ee099050bcda5: Sending DeleteTablet for 3 replicas of tablet b80efd9c02dc43bb9bc2e8598a137ba7
I20260926 20:19:50.752795 17615 tablet_service.cc:1514] Processing DeleteTablet for tablet b80efd9c02dc43bb9bc2e8598a137ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:50 UTC) from {username='slave'} at 127.0.0.1:46074
I20260926 20:19:50.752832 17484 tablet_service.cc:1514] Processing DeleteTablet for tablet b80efd9c02dc43bb9bc2e8598a137ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:50 UTC) from {username='slave'} at 127.0.0.1:33000
I20260926 20:19:50.753427 17851 tablet_replica.cc:330] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b: stopping tablet replica
I20260926 20:19:50.753580 17746 tablet_service.cc:1514] Processing DeleteTablet for tablet b80efd9c02dc43bb9bc2e8598a137ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:50 UTC) from {username='slave'} at 127.0.0.1:57776
I20260926 20:19:50.753806 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 17422
I20260926 20:19:50.753819 17851 raft_consensus.cc:2229] T b80efd9c02dc43bb9bc2e8598a137ba7 P ec229d6ccfbd4b4ea00f387da9138a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:50.754415 17852 tablet_replica.cc:330] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: stopping tablet replica
I20260926 20:19:50.754753 17852 raft_consensus.cc:2229] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:50.755054 17852 raft_consensus.cc:2258] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:50.756469 17852 ts_tablet_manager.cc:1905] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:50.759919 17852 ts_tablet_manager.cc:1918] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:50.760048 17852 log.cc:1196] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/b80efd9c02dc43bb9bc2e8598a137ba7
I20260926 20:19:50.760406 17852 ts_tablet_manager.cc:1939] T b80efd9c02dc43bb9bc2e8598a137ba7 P e1486b2ecf4a4ad19c91b9b61946c700: Deleting consensus metadata
I20260926 20:19:50.762163 17347 catalog_manager.cc:4841] TS e1486b2ecf4a4ad19c91b9b61946c700 (127.31.119.194:43341): tablet b80efd9c02dc43bb9bc2e8598a137ba7 (table test-table [id=cdfcc5710b0e4d7cbf12636ed2d597ba]) successfully deleted
W20260926 20:19:50.763293 17346 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:36839 (error 108)
I20260926 20:19:50.764402 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:50.764084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:36839 (local address 127.0.0.1:33014)
0926 20:19:50.764221 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:50.764297 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:36839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 20:19:50.764516 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 17553
W20260926 20:19:50.764855 17346 catalog_manager.cc:4584] TS ec229d6ccfbd4b4ea00f387da9138a6b (127.31.119.193:36839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b80efd9c02dc43bb9bc2e8598a137ba7: Network error: Client connection negotiation failed: client connection to 127.31.119.193:36839: connect: Connection refused (error 111)
I20260926 20:19:50.769344 17853 tablet_replica.cc:330] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3: stopping tablet replica
I20260926 20:19:50.769747 17853 raft_consensus.cc:2229] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:19:50.770176 17853 raft_consensus.cc:2258] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:50.771713 17853 ts_tablet_manager.cc:1905] T b80efd9c02dc43bb9bc2e8598a137ba7 P 543630ad31a941ca9962ede24e601aa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:50.772215 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 17684
I20260926 20:19:50.778239 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 17331
2026-09-26T20:19:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1250 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:19:50 dist-test-slave-vnbh krb5kdc[17857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:19:50 dist-test-slave-vnbh krb5kdc[17857](info): set up 2 sockets
Sep 26 20:19:50 dist-test-slave-vnbh krb5kdc[17857](info): commencing operation
krb5kdc: starting...
W20260926 20:19:51.826124 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:19:51 dist-test-slave-vnbh krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:19:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:19:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:19:51.957019 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:40605
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:40605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:52.069013 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:52.069412 17870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:52.069525 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:52.073809 17870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:19:52.073923 17870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:52.073966 17870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:52.074002 17870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:19:52.074031 17870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:19:52.079485 17870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:40605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:40605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:52.080667 17870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:52.081812 17870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:52.090736 17876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:52.090834 17875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:52.090986 17870 server_base.cc:1034] running on GCE node
W20260926 20:19:52.090754 17878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:52.091784 17870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:52.093017 17870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:52.094199 17870 hybrid_clock.cc:648] HybridClock initialized: now 1790453992094178 us; error 32 us; skew 500 ppm
Sep 26 20:19:52 dist-test-slave-vnbh krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453992, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:52.097031 17870 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:19:52.097978 17870 webserver.cc:460] Webserver started at http://127.31.119.254:45039/ using document root <none> and password file <none>
I20260926 20:19:52.098552 17870 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:52.098650 17870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:52.098897 17870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:52.100967 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "86eadfa1ce9ff289013cd69e164a5e50"
server_key_iv: "55acc780ebf191f0accc378d9b9d4d90"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.101569 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "86eadfa1ce9ff289013cd69e164a5e50"
server_key_iv: "55acc780ebf191f0accc378d9b9d4d90"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.105530 17870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:19:52.108224 17885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:52.109642 17870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:52.109792 17870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "86eadfa1ce9ff289013cd69e164a5e50"
server_key_iv: "55acc780ebf191f0accc378d9b9d4d90"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.109918 17870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:52.137876 17870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:52.142040 17870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:52.142213 17870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:52.149914 17870 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:40605
I20260926 20:19:52.149933 17937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:40605 every 8 connection(s)
I20260926 20:19:52.150960 17870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:19:52.153970 17938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:52.154230 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 17870
I20260926 20:19:52.154407 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:19:52.154731 32223 external_mini_cluster.cc:1428] Setting key acc0f58be4b5d8a32b16fcb43c60747a
I20260926 20:19:52.160770 17938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1: Bootstrap starting.
I20260926 20:19:52.163204 17938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:52.163972 17938 log.cc:824] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:19:52 dist-test-slave-vnbh krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:52.166373 17938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1: No bootstrap required, opened a new log
I20260926 20:19:52.170805 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.156226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.31.119.254:40605)
0926 20:19:52.156692 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:52.156707 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:52.157370 (+ 663us) server_negotiation.cc:407] Connection header received
0926 20:19:52.158402 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:52.158422 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:52.158779 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:52.159193 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:52.160027 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.161109 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.162063 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.162559 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.162748 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.165665 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:52.165671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:52.165685 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:52.165712 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:52.167677 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:52.168189 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:52.168194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:52.168199 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:52.168312 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:52.168672 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:52.168675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:52.168676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:52.169089 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:52.169241 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:52.169599 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 20:19:52.169934 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":123,"threads_started":1}
I20260926 20:19:52.170774 17938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40605 } }
I20260926 20:19:52.171079 17938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:52.171154 17938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf14e73e6ca46e5a46b7c3071ac10f1, State: Initialized, Role: FOLLOWER
I20260926 20:19:52.171614 17938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40605 } }
I20260926 20:19:52.171871 17938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:19:52.171967 17938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:19:52.172129 17938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:52.173091 17938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40605 } }
I20260926 20:19:52.173480 17938 leader_election.cc:302] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bf14e73e6ca46e5a46b7c3071ac10f1; no voters:
I20260926 20:19:52.173813 17938 leader_election.cc:288] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:19:52.174049 17943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:52.174322 17943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf14e73e6ca46e5a46b7c3071ac10f1, State: Running, Role: LEADER
I20260926 20:19:52.174655 17943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40605 } }
I20260926 20:19:52.175235 17938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:19:52.176538 17945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bf14e73e6ca46e5a46b7c3071ac10f1. Latest consensus state: current_term: 1 leader_uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40605 } } }
I20260926 20:19:52.176666 17945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:52.176994 17944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14e73e6ca46e5a46b7c3071ac10f1" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40605 } } }
I20260926 20:19:52.177085 17944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1 [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:52.177244 17952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:19:52.180053 17952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:19:52.188234 17952 catalog_manager.cc:1347] Generated new cluster ID: 817b3a66b5ab46829ecab551889d21df
I20260926 20:19:52.188333 17952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:19:52.211117 17952 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:19:52.212407 17952 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:19:52.219273 17952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1: Generated new TSK 0
I20260926 20:19:52.220036 17952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:19:52.253633 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40605
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:52.385555 17964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:52.385855 17964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:52.385903 17964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:52.389938 17964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:52.389999 17964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:52.390059 17964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:19:52.395349 17964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:52.396386 17964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:52.397395 17964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:52.407244 17969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:52.407248 17970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:52.407457 17964 server_base.cc:1034] running on GCE node
W20260926 20:19:52.407248 17972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:52.407961 17964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:52.408630 17964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:52.409835 17964 hybrid_clock.cc:648] HybridClock initialized: now 1790453992409831 us; error 52 us; skew 500 ppm
Sep 26 20:19:52 dist-test-slave-vnbh krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453992, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:52.412834 17964 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:19:52.413866 17964 webserver.cc:460] Webserver started at http://127.31.119.193:43885/ using document root <none> and password file <none>
I20260926 20:19:52.414708 17964 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:52.414814 17964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:52.415053 17964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:52.417246 17964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "a5936b6c3f004feebf16632833e892c0"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "a8a27f0230bbf65f17d2d17dbf960d6c"
server_key_iv: "445ba62ceb7e1da594b1eebfea752c3b"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.417951 17964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "a5936b6c3f004feebf16632833e892c0"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "a8a27f0230bbf65f17d2d17dbf960d6c"
server_key_iv: "445ba62ceb7e1da594b1eebfea752c3b"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.422032 17964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:19:52.424672 17979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:52.426067 17964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:52.426203 17964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "a5936b6c3f004feebf16632833e892c0"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "a8a27f0230bbf65f17d2d17dbf960d6c"
server_key_iv: "445ba62ceb7e1da594b1eebfea752c3b"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.426319 17964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:52.439072 17964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:52.442685 17964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:52.442917 17964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:52.443573 17964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:52.444465 17964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:52.444514 17964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:52.444589 17964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:52.444620 17964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:52.454546 17964 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:35775
I20260926 20:19:52.454564 18092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:35775 every 8 connection(s)
I20260926 20:19:52.455497 17964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:19:52.461896 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 17964
I20260926 20:19:52.462080 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:19:52.462424 32223 external_mini_cluster.cc:1428] Setting key 828855281a91dc753df8fb5795bc2746
Sep 26 20:19:52 dist-test-slave-vnbh krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453992, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:52.471560 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.457518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:35909 (local address 127.31.119.254:40605)
0926 20:19:52.457668 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:52.457673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:52.458592 (+ 919us) server_negotiation.cc:407] Connection header received
0926 20:19:52.459935 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:52.459938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:52.459989 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:52.460068 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:52.461279 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.461874 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.463489 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.463847 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.463968 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.467490 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:52.467494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:52.467496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:52.467525 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:52.469197 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:52.469809 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:52.469813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:52.469815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:52.469907 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:52.470291 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:52.470297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:52.470302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:52.470513 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:52.470636 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:52.471251 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 20:19:52.471390 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:19:52.472496 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.457950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40605 (local address 127.31.119.193:35909)
0926 20:19:52.458420 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:52.458464 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:52.459660 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:52.460251 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:52.460258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:52.460638 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:52.461128 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.461142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.462022 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.462027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.463365 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.463375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.464039 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.464043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.464240 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.464972 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:52.464999 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:52.467247 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:52.469368 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:52.469375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:52.469394 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:52.469684 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:52.470022 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:52.470025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:52.470027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:52.470183 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:52.470633 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:52.470640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:52.470926 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 20:19:52.471197 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 20:19:52.471572 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
I20260926 20:19:52.474035 18093 heartbeater.cc:347] Connected to a master server at 127.31.119.254:40605
I20260926 20:19:52.474332 18093 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:52.474995 18093 heartbeater.cc:511] Master 127.31.119.254:40605 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
I20260926 20:19:52.476754 17902 ts_manager.cc:194] Registered new tserver with Master: a5936b6c3f004feebf16632833e892c0 (127.31.119.193:35775)
I20260926 20:19:52.478037 17902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:35909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:19:52.497620 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40605
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:52.619372 18098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:52.619668 18098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:52.619715 18098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:52.624768 18098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:52.624836 18098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:52.624904 18098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:19:52.631177 18098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:52.632279 18098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:52.633394 18098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:52.643646 18103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:52.643731 18098 server_base.cc:1034] running on GCE node
W20260926 20:19:52.643700 18104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:52.643662 18106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:52.644342 18098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:52.645052 18098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:52.646239 18098 hybrid_clock.cc:648] HybridClock initialized: now 1790453992646213 us; error 37 us; skew 500 ppm
Sep 26 20:19:52 dist-test-slave-vnbh krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453992, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:52.649014 18098 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:19:52.649983 18098 webserver.cc:460] Webserver started at http://127.31.119.194:42571/ using document root <none> and password file <none>
I20260926 20:19:52.650525 18098 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:52.650584 18098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:52.650746 18098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:52.652786 18098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "de1ee1b19ce44032b94830a15f28cfc6"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "dcbee9ebb6dd8fa182584e4b3455ed5d"
server_key_iv: "5b79da8c33bc2d0efa52a4d5a6795e11"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.653388 18098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "de1ee1b19ce44032b94830a15f28cfc6"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "dcbee9ebb6dd8fa182584e4b3455ed5d"
server_key_iv: "5b79da8c33bc2d0efa52a4d5a6795e11"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.657452 18098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 20:19:52.660984 18113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:52.662472 18098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:52.662591 18098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "de1ee1b19ce44032b94830a15f28cfc6"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "dcbee9ebb6dd8fa182584e4b3455ed5d"
server_key_iv: "5b79da8c33bc2d0efa52a4d5a6795e11"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.662693 18098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:52.696107 18098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:52.699506 18098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:52.699666 18098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:52.700318 18098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:52.701139 18098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:52.701184 18098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:52.701224 18098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:52.701241 18098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:52.710749 18098 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:37629
I20260926 20:19:52.710774 18226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:37629 every 8 connection(s)
I20260926 20:19:52.711689 18098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:19:52.715723 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 18098
I20260926 20:19:52.715831 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:19:52.716130 32223 external_mini_cluster.cc:1428] Setting key f694c3c19cf7a58ba87264611e7fc777
Sep 26 20:19:52 dist-test-slave-vnbh krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453992, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
I20260926 20:19:52.729230 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.713756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:43805 (local address 127.31.119.254:40605)
0926 20:19:52.714005 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:52.714008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:52.714876 (+ 868us) server_negotiation.cc:407] Connection header received
0926 20:19:52.716389 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:52.716393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:52.716455 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:52.716586 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:52.718073 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.718634 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.720540 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.720784 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.720882 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.724638 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:52.724641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:52.724643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:52.724695 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:52.726460 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:52.727320 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:52.727323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:52.727324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:52.727418 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:52.727955 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:52.727958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:52.727959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:52.728169 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:52.728259 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:52.728981 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 20:19:52.729096 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260926 20:19:52.730463 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.714128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40605 (local address 127.31.119.194:43805)
0926 20:19:52.714668 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:52.714713 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:52.716113 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:52.716776 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:52.716783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:52.717351 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:52.717923 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.717938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.718775 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.718779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.720263 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.720277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.720964 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.720968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.721140 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.721988 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:52.722007 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:52.724190 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:52.726625 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:52.726632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:52.726658 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:52.727156 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:52.727549 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:52.727555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:52.727561 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:52.727815 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:52.728315 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:52.728321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:52.728606 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 20:19:52.728921 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 20:19:52.729230 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":153,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
I20260926 20:19:52.731791 18227 heartbeater.cc:347] Connected to a master server at 127.31.119.254:40605
I20260926 20:19:52.732062 18227 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:52.732605 18227 heartbeater.cc:511] Master 127.31.119.254:40605 requested a full tablet report, sending...
I20260926 20:19:52.733870 17902 ts_manager.cc:194] Registered new tserver with Master: de1ee1b19ce44032b94830a15f28cfc6 (127.31.119.194:37629)
I20260926 20:19:52.734535 17902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:43805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:19:52.749482 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40605
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:52.869742 18232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:52.870041 18232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:52.870087 18232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:52.874320 18232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:52.874392 18232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:52.874459 18232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:19:52.880260 18232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:40605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:52.881378 18232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:52.882418 18232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:52.892814 18240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:52.892814 18237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:52.892814 18238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:52.893239 18232 server_base.cc:1034] running on GCE node
I20260926 20:19:52.893705 18232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:52.894445 18232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:52.895622 18232 hybrid_clock.cc:648] HybridClock initialized: now 1790453992895611 us; error 36 us; skew 500 ppm
Sep 26 20:19:52 dist-test-slave-vnbh krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453992, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:52.898747 18232 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:19:52.899664 18232 webserver.cc:460] Webserver started at http://127.31.119.195:35931/ using document root <none> and password file <none>
I20260926 20:19:52.900278 18232 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:52.900352 18232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:52.900609 18232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:52.902932 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "c44fa2e43c1b48b3883700e3f6db27dd"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "1860355f6a5d40f561e6f6b73743a8da"
server_key_iv: "99643fc72a5e7797f2aabe460a07d49a"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.903537 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "c44fa2e43c1b48b3883700e3f6db27dd"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "1860355f6a5d40f561e6f6b73743a8da"
server_key_iv: "99643fc72a5e7797f2aabe460a07d49a"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.907632 18232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:19:52.910490 18247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:52.911931 18232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:52.912118 18232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "c44fa2e43c1b48b3883700e3f6db27dd"
format_stamp: "Formatted at 2026-09-26 20:19:52 on dist-test-slave-vnbh"
server_key: "1860355f6a5d40f561e6f6b73743a8da"
server_key_iv: "99643fc72a5e7797f2aabe460a07d49a"
server_key_version: "encryptionkey@0"
I20260926 20:19:52.912302 18232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:52.925617 18232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:52.929520 18232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:52.929750 18232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:52.930480 18232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:52.931370 18232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:52.931455 18232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:52.931527 18232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:52.931576 18232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:52.940518 18232 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:39871
I20260926 20:19:52.940546 18360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:39871 every 8 connection(s)
I20260926 20:19:52.941558 18232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:19:52.946945 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 18232
I20260926 20:19:52.947053 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:19:52.947364 32223 external_mini_cluster.cc:1428] Setting key 324a1f7540776adf4bccdc9d1d6982f0
Sep 26 20:19:52 dist-test-slave-vnbh krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453992, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:52.956246 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.943546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:42861 (local address 127.31.119.254:40605)
0926 20:19:52.943701 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:52.943704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:52.944501 (+ 797us) server_negotiation.cc:407] Connection header received
0926 20:19:52.945858 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:52.945861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:52.945908 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:52.945988 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:52.947340 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.947942 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.949127 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.949452 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.949575 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.952337 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:52.952341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:52.952343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:52.952370 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:52.953771 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:52.954562 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:52.954566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:52.954567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:52.954654 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:52.955130 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:52.955133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:52.955135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:52.955338 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:52.955430 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:52.956004 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 20:19:52.956117 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:19:52.957268 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.943860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:40605 (local address 127.31.119.195:42861)
0926 20:19:52.944326 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:52.944378 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:52.945643 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:52.946152 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:52.946158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:52.946511 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:52.947170 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.947187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.948082 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.948086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.948987 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.948996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.949615 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.949620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.949803 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.950504 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:52.950520 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:52.952132 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:52.953971 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:52.953977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:52.953992 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:52.954283 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:52.954780 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:52.954782 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:52.954784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:52.954987 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:52.955474 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:52.955480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:52.955755 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 20:19:52.955943 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 20:19:52.956256 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260926 20:19:52.958865 18361 heartbeater.cc:347] Connected to a master server at 127.31.119.254:40605
I20260926 20:19:52.959223 18361 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:52.959838 18361 heartbeater.cc:511] Master 127.31.119.254:40605 requested a full tablet report, sending...
I20260926 20:19:52.961123 17902 ts_manager.cc:194] Registered new tserver with Master: c44fa2e43c1b48b3883700e3f6db27dd (127.31.119.195:39871)
I20260926 20:19:52.961798 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:19:52.961819 17902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:42861
I20260926 20:19:52.973726 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.964107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.31.119.254:40605)
0926 20:19:52.964279 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:52.964285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:52.964441 (+ 156us) server_negotiation.cc:407] Connection header received
0926 20:19:52.964628 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:52.964632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:52.964715 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:52.964818 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:52.965757 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.966442 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.967572 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.967837 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.967937 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.969828 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:52.969831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:52.969833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:52.969859 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:52.971366 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:52.971960 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:52.971963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:52.971964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:52.972044 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:52.972499 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:52.972502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:52.972504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:52.972813 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:52.972951 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:52.973414 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 20:19:52.973545 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:19:52.977592 17902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:19:52.980818 17902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:19:52.996728 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.991265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.31.119.195:39871)
0926 20:19:52.992319 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:52.992332 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:52.992355 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:19:52.992424 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:52.992429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:52.992606 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:52.992749 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:52.992998 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.994066 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.995314 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.995920 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.996091 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.996246 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:52.996452 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 20:19:52.996559 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":140,"threads_started":1}
I20260926 20:19:52.996722 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.990816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:39871 (local address 127.0.0.1:40426)
0926 20:19:52.992175 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:52.992189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:52.992265 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:52.992768 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:52.992772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:52.992791 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:52.992867 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.992872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.994217 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.994222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.995120 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.995131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.996099 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.996102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.996220 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.996260 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:19:52.996418 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 20:19:52.996564 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260926 20:19:52.997030 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.990293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.31.119.193:35775)
0926 20:19:52.990829 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:52.990838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:52.991445 (+ 607us) server_negotiation.cc:407] Connection header received
0926 20:19:52.991536 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:52.991544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:52.991756 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:52.991880 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:52.992296 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.993431 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.995291 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.996095 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.996199 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.996304 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:52.996771 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 20:19:52.996885 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":137,"threads_started":1}
I20260926 20:19:52.997620 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.990303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.31.119.194:37629)
0926 20:19:52.990961 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:52.990975 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:52.991567 (+ 592us) server_negotiation.cc:407] Connection header received
0926 20:19:52.991652 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:52.991656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:52.991806 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:52.991988 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:52.992301 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.993225 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.995115 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.995998 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.996133 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.996273 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:52.997356 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0926 20:19:52.997475 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":132,"threads_started":1}
I20260926 20:19:52.996920 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.990051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:37629 (local address 127.0.0.1:47868)
0926 20:19:52.991327 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:52.991341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:52.991466 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:52.991993 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:52.991997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:52.992015 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:52.992155 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.992161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.993396 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.993400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.994970 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.994981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.996357 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.996361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.996560 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.996604 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:19:52.996675 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:19:52.996780 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1206,"mutex_wait_us":719,"thread_start_us":114,"threads_started":1}
I20260926 20:19:52.998379 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:52.990051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35775 (local address 127.0.0.1:46406)
0926 20:19:52.991076 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:52.991133 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:52.991282 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:52.991914 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:52.991923 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:52.991950 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:52.992152 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.992163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.993546 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.993550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.994970 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:52.994981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:52.996357 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:52.996361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:52.996555 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:52.996602 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:19:52.996675 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:19:52.997238 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":618,"thread_start_us":85,"threads_started":1}
I20260926 20:19:53.000437 18295 tablet_service.cc:1467] Processing CreateTablet for tablet ec0716f6113745bd81f482d668406d0d (DEFAULT_TABLE table=test-table [id=95147dea57de44358e2bc289a3f16173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:53.000698 18161 tablet_service.cc:1467] Processing CreateTablet for tablet ec0716f6113745bd81f482d668406d0d (DEFAULT_TABLE table=test-table [id=95147dea57de44358e2bc289a3f16173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:53.000907 18027 tablet_service.cc:1467] Processing CreateTablet for tablet ec0716f6113745bd81f482d668406d0d (DEFAULT_TABLE table=test-table [id=95147dea57de44358e2bc289a3f16173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:53.001554 18295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec0716f6113745bd81f482d668406d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:53.001789 18161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec0716f6113745bd81f482d668406d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:53.001794 18027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec0716f6113745bd81f482d668406d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:53.009713 18378 tablet_bootstrap.cc:492] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: Bootstrap starting.
I20260926 20:19:53.011266 18379 tablet_bootstrap.cc:492] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0: Bootstrap starting.
I20260926 20:19:53.012409 18378 tablet_bootstrap.cc:654] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:53.013458 18379 tablet_bootstrap.cc:654] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:53.013774 18378 log.cc:824] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:53.014605 18379 log.cc:824] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:53.015199 18380 tablet_bootstrap.cc:492] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: Bootstrap starting.
I20260926 20:19:53.016275 18378 tablet_bootstrap.cc:492] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: No bootstrap required, opened a new log
I20260926 20:19:53.016525 18378 ts_tablet_manager.cc:1397] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 20:19:53.016788 18379 tablet_bootstrap.cc:492] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0: No bootstrap required, opened a new log
I20260926 20:19:53.017025 18379 ts_tablet_manager.cc:1397] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 20:19:53.017845 18380 tablet_bootstrap.cc:654] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:53.019183 18380 log.cc:824] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:53.020905 18378 raft_consensus.cc:348] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 } } peers { permanent_uuid: "de1ee1b19ce44032b94830a15f28cfc6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37629 } } peers { permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 } }
I20260926 20:19:53.021215 18378 raft_consensus.cc:374] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:53.021274 18378 raft_consensus.cc:729] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c44fa2e43c1b48b3883700e3f6db27dd, State: Initialized, Role: FOLLOWER
I20260926 20:19:53.021092 18379 raft_consensus.cc:348] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 } } peers { permanent_uuid: "de1ee1b19ce44032b94830a15f28cfc6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37629 } } peers { permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 } }
I20260926 20:19:53.021404 18379 raft_consensus.cc:374] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:53.021471 18379 raft_consensus.cc:729] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5936b6c3f004feebf16632833e892c0, State: Initialized, Role: FOLLOWER
I20260926 20:19:53.021703 18380 tablet_bootstrap.cc:492] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: No bootstrap required, opened a new log
I20260926 20:19:53.021905 18380 ts_tablet_manager.cc:1397] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:19:53.021826 18378 consensus_queue.cc:260] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 } } peers { permanent_uuid: "de1ee1b19ce44032b94830a15f28cfc6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37629 } } peers { permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 } }
I20260926 20:19:53.021955 18379 consensus_queue.cc:260] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 } } peers { permanent_uuid: "de1ee1b19ce44032b94830a15f28cfc6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37629 } } peers { permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 } }
I20260926 20:19:53.022856 18379 ts_tablet_manager.cc:1428] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:19:53.022910 18378 ts_tablet_manager.cc:1428] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260926 20:19:53.023262 18361 heartbeater.cc:503] Master 127.31.119.254:40605 was elected leader, sending a full tablet report...
I20260926 20:19:53.023264 18093 heartbeater.cc:503] Master 127.31.119.254:40605 was elected leader, sending a full tablet report...
I20260926 20:19:53.026074 18380 raft_consensus.cc:348] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 } } peers { permanent_uuid: "de1ee1b19ce44032b94830a15f28cfc6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37629 } } peers { permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 } }
I20260926 20:19:53.026448 18380 raft_consensus.cc:374] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:53.026511 18380 raft_consensus.cc:729] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1ee1b19ce44032b94830a15f28cfc6, State: Initialized, Role: FOLLOWER
I20260926 20:19:53.027000 18380 consensus_queue.cc:260] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 } } peers { permanent_uuid: "de1ee1b19ce44032b94830a15f28cfc6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37629 } } peers { permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 } }
I20260926 20:19:53.027768 18227 heartbeater.cc:503] Master 127.31.119.254:40605 was elected leader, sending a full tablet report...
I20260926 20:19:53.028204 18380 ts_tablet_manager.cc:1428] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:19:53.085558 18386 raft_consensus.cc:482] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:19:53.085780 18386 raft_consensus.cc:504] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 } } peers { permanent_uuid: "de1ee1b19ce44032b94830a15f28cfc6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37629 } } peers { permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 } }
I20260926 20:19:53.087208 18386 leader_election.cc:288] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5936b6c3f004feebf16632833e892c0 (127.31.119.193:35775), c44fa2e43c1b48b3883700e3f6db27dd (127.31.119.195:39871)
I20260926 20:19:53.091302 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.087480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35775 (local address 127.31.119.194:45459)
0926 20:19:53.087676 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:53.087695 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:53.087814 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:53.088136 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:53.088140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:53.088162 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:53.088308 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:53.088318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.089187 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.089193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:53.090001 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:53.090010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.090820 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.090823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:53.090939 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:53.090964 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:19:53.091061 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:19:53.091163 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":1}
I20260926 20:19:53.091344 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.087584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:45459 (local address 127.31.119.193:35775)
0926 20:19:53.087815 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:53.087820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:53.087841 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:19:53.087907 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:53.087912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:53.087971 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:53.088049 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:53.088414 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.089072 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.090158 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.090697 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.090834 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:53.090892 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:53.091163 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 20:19:53.091240 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:19:53.091742 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.087790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:39871 (local address 127.31.119.194:45327)
0926 20:19:53.088172 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:53.088185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:53.088279 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:53.088519 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:53.088522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:53.088539 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:53.088660 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:53.088665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.089449 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.089454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:53.090295 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:53.090305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.091073 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.091076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:53.091260 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:53.091282 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:19:53.091531 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 20:19:53.091602 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":48,"thread_start_us":128,"threads_started":1}
I20260926 20:19:53.091789 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.087908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:45327 (local address 127.31.119.195:39871)
0926 20:19:53.088059 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:53.088066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:53.088283 (+ 217us) server_negotiation.cc:407] Connection header received
0926 20:19:53.088354 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:53.088357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:53.088406 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:53.088486 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:53.088777 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.089260 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.090416 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.090945 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.091065 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:53.091122 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:53.091586 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 20:19:53.091675 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:19:53.092146 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec0716f6113745bd81f482d668406d0d" candidate_uuid: "de1ee1b19ce44032b94830a15f28cfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5936b6c3f004feebf16632833e892c0" is_pre_election: true
I20260926 20:19:53.092494 18047 raft_consensus.cc:2454] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1ee1b19ce44032b94830a15f28cfc6 in term 0.
I20260926 20:19:53.092507 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec0716f6113745bd81f482d668406d0d" candidate_uuid: "de1ee1b19ce44032b94830a15f28cfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" is_pre_election: true
I20260926 20:19:53.092789 18315 raft_consensus.cc:2454] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de1ee1b19ce44032b94830a15f28cfc6 in term 0.
I20260926 20:19:53.093047 18115 leader_election.cc:302] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5936b6c3f004feebf16632833e892c0, de1ee1b19ce44032b94830a15f28cfc6; no voters:
I20260926 20:19:53.093333 18386 raft_consensus.cc:2781] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:19:53.093420 18386 raft_consensus.cc:482] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:19:53.093452 18386 raft_consensus.cc:3037] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:53.094523 18386 raft_consensus.cc:504] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 } } peers { permanent_uuid: "de1ee1b19ce44032b94830a15f28cfc6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37629 } } peers { permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 } }
I20260926 20:19:53.094887 18386 leader_election.cc:288] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [CANDIDATE]: Term 1 election: Requested vote from peers a5936b6c3f004feebf16632833e892c0 (127.31.119.193:35775), c44fa2e43c1b48b3883700e3f6db27dd (127.31.119.195:39871)
I20260926 20:19:53.095274 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec0716f6113745bd81f482d668406d0d" candidate_uuid: "de1ee1b19ce44032b94830a15f28cfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5936b6c3f004feebf16632833e892c0"
I20260926 20:19:53.095321 18315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec0716f6113745bd81f482d668406d0d" candidate_uuid: "de1ee1b19ce44032b94830a15f28cfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c44fa2e43c1b48b3883700e3f6db27dd"
I20260926 20:19:53.095384 18047 raft_consensus.cc:3037] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:53.095498 18315 raft_consensus.cc:3037] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:53.096573 18047 raft_consensus.cc:2454] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1ee1b19ce44032b94830a15f28cfc6 in term 1.
I20260926 20:19:53.096599 18315 raft_consensus.cc:2454] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de1ee1b19ce44032b94830a15f28cfc6 in term 1.
I20260926 20:19:53.096912 18115 leader_election.cc:302] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5936b6c3f004feebf16632833e892c0, de1ee1b19ce44032b94830a15f28cfc6; no voters:
I20260926 20:19:53.097086 18386 raft_consensus.cc:2781] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:53.097386 18386 raft_consensus.cc:686] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 1 LEADER]: Becoming Leader. State: Replica: de1ee1b19ce44032b94830a15f28cfc6, State: Running, Role: LEADER
I20260926 20:19:53.097740 18386 consensus_queue.cc:237] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 } } peers { permanent_uuid: "de1ee1b19ce44032b94830a15f28cfc6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37629 } } peers { permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 } }
I20260926 20:19:53.101071 17902 catalog_manager.cc:5495] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to de1ee1b19ce44032b94830a15f28cfc6 (127.31.119.194). New cstate: current_term: 1 leader_uuid: "de1ee1b19ce44032b94830a15f28cfc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de1ee1b19ce44032b94830a15f28cfc6" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 } health_report { overall_health: UNKNOWN } } }
I20260926 20:19:53.125097 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.120896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.31.119.194:37629)
0926 20:19:53.121077 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:53.121082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:53.121099 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:19:53.121362 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:53.121365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:53.121441 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:53.121569 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:19:53.121949 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.122472 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.123382 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.123630 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.123778 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:53.124120 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:19:53.124602 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:19:53.124806 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:53.124894 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 20:19:53.124933 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 20:19:53.131611 18047 raft_consensus.cc:1261] T ec0716f6113745bd81f482d668406d0d P a5936b6c3f004feebf16632833e892c0 [term 1 FOLLOWER]: Refusing update from remote peer de1ee1b19ce44032b94830a15f28cfc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:53.131611 18315 raft_consensus.cc:1261] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd [term 1 FOLLOWER]: Refusing update from remote peer de1ee1b19ce44032b94830a15f28cfc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:53.132416 18386 consensus_queue.cc:1035] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c44fa2e43c1b48b3883700e3f6db27dd" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:53.132638 18388 consensus_queue.cc:1035] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5936b6c3f004feebf16632833e892c0" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:53.141459 18392 mvcc.cc:205] Tried to move back new op lower bound from 7333699555858599936 to 7333699555731111936. Current Snapshot: MvccSnapshot[applied={T|T < 7333699555858599936}]
I20260926 20:19:53.142398 18393 mvcc.cc:205] Tried to move back new op lower bound from 7333699555858599936 to 7333699555731111936. Current Snapshot: MvccSnapshot[applied={T|T < 7333699555858599936}]
I20260926 20:19:53.149708 17902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:53.149937 17902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:53.152845 17902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bf14e73e6ca46e5a46b7c3071ac10f1: Sending DeleteTablet for 3 replicas of tablet ec0716f6113745bd81f482d668406d0d
I20260926 20:19:53.153756 18161 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0716f6113745bd81f482d668406d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:53 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:47868
I20260926 20:19:53.154306 18401 tablet_replica.cc:330] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: stopping tablet replica
I20260926 20:19:53.154309 18027 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0716f6113745bd81f482d668406d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:53 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:46406
I20260926 20:19:53.154814 18401 raft_consensus.cc:2229] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:19:53.155259 18401 raft_consensus.cc:2258] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:53.155242 18295 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0716f6113745bd81f482d668406d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:53 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:40426
I20260926 20:19:53.156602 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 17964
I20260926 20:19:53.157176 18401 ts_tablet_manager.cc:1905] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:53.160243 18401 ts_tablet_manager.cc:1918] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:53.160475 18401 log.cc:1196] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/ec0716f6113745bd81f482d668406d0d
I20260926 20:19:53.160900 18401 ts_tablet_manager.cc:1939] T ec0716f6113745bd81f482d668406d0d P de1ee1b19ce44032b94830a15f28cfc6: Deleting consensus metadata
I20260926 20:19:53.161221 18403 tablet_replica.cc:330] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: stopping tablet replica
I20260926 20:19:53.162441 18403 raft_consensus.cc:2229] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:53.163647 17886 catalog_manager.cc:4841] TS de1ee1b19ce44032b94830a15f28cfc6 (127.31.119.194:37629): tablet ec0716f6113745bd81f482d668406d0d (table test-table [id=95147dea57de44358e2bc289a3f16173]) successfully deleted
I20260926 20:19:53.164234 18403 raft_consensus.cc:2258] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:53.166013 18403 ts_tablet_manager.cc:1905] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:19:53.168336 17887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:35775 (error 108)
I20260926 20:19:53.169204 18403 ts_tablet_manager.cc:1918] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:53.169446 18403 log.cc:1196] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/ec0716f6113745bd81f482d668406d0d
I20260926 20:19:53.169776 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 18098
I20260926 20:19:53.169829 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.169506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35775 (local address 127.0.0.1:46410)
0926 20:19:53.169648 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:53.169729 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:35775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:19:53.171106 18403 ts_tablet_manager.cc:1939] T ec0716f6113745bd81f482d668406d0d P c44fa2e43c1b48b3883700e3f6db27dd: Deleting consensus metadata
W20260926 20:19:53.173623 17887 catalog_manager.cc:4584] TS a5936b6c3f004feebf16632833e892c0 (127.31.119.193:35775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec0716f6113745bd81f482d668406d0d: Network error: Client connection negotiation failed: client connection to 127.31.119.193:35775: connect: Connection refused (error 111)
I20260926 20:19:53.173905 17887 catalog_manager.cc:4841] TS c44fa2e43c1b48b3883700e3f6db27dd (127.31.119.195:39871): tablet ec0716f6113745bd81f482d668406d0d (table test-table [id=95147dea57de44358e2bc289a3f16173]) successfully deleted
I20260926 20:19:53.177754 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 18232
I20260926 20:19:53.184075 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 17870
2026-09-26T20:19:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2407 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T20:19:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:19:53Z Disabled control of system clock
I20260926 20:19:53.216988 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:44695
--webserver_interface=127.31.119.254
--webserver_port=0
--builtin_ntp_servers=127.31.119.212:44845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:44695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:19:53.336453 18408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:53.336758 18408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:53.336796 18408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:53.340998 18408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:19:53.341086 18408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:53.341101 18408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:53.341120 18408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:19:53.341137 18408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:19:53.346434 18408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:44845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:44695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:53.347590 18408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:53.349017 18408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:53.358428 18413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:53.358407 18416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:53.358413 18414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:53.358791 18408 server_base.cc:1034] running on GCE node
I20260926 20:19:53.359541 18408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:53.360750 18408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:53.361944 18408 hybrid_clock.cc:648] HybridClock initialized: now 1790453993361913 us; error 43 us; skew 500 ppm
I20260926 20:19:53.364145 18408 webserver.cc:460] Webserver started at http://127.31.119.254:33609/ using document root <none> and password file <none>
I20260926 20:19:53.364833 18408 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:53.364964 18408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:53.365223 18408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:53.367477 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "559328ed5ecf409798f0b5acf79058b4"
format_stamp: "Formatted at 2026-09-26 20:19:53 on dist-test-slave-vnbh"
server_key: "d7bac01be367c641bb340ba4d3114ba3"
server_key_iv: "fda2ab73c753a22678a7fd0d46af4d22"
server_key_version: "encryptionkey@0"
I20260926 20:19:53.368098 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "559328ed5ecf409798f0b5acf79058b4"
format_stamp: "Formatted at 2026-09-26 20:19:53 on dist-test-slave-vnbh"
server_key: "d7bac01be367c641bb340ba4d3114ba3"
server_key_iv: "fda2ab73c753a22678a7fd0d46af4d22"
server_key_version: "encryptionkey@0"
I20260926 20:19:53.372087 18408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:19:53.375092 18422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:53.376662 18408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:19:53.376855 18408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "559328ed5ecf409798f0b5acf79058b4"
format_stamp: "Formatted at 2026-09-26 20:19:53 on dist-test-slave-vnbh"
server_key: "d7bac01be367c641bb340ba4d3114ba3"
server_key_iv: "fda2ab73c753a22678a7fd0d46af4d22"
server_key_version: "encryptionkey@0"
I20260926 20:19:53.377005 18408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:53.386920 18408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:53.387650 18408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:53.387861 18408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:53.395277 18408 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:44695
I20260926 20:19:53.395305 18474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:44695 every 8 connection(s)
I20260926 20:19:53.396286 18408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:19:53.399616 18475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:53.404036 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 18408
I20260926 20:19:53.404148 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:19:53.404479 32223 external_mini_cluster.cc:1428] Setting key fd90ea31c94dec6b911e218ef93b6189
I20260926 20:19:53.405850 18475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4: Bootstrap starting.
I20260926 20:19:53.408627 18475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:53.409622 18475 log.cc:824] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:53.411810 18475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4: No bootstrap required, opened a new log
I20260926 20:19:53.415953 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.406216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.31.119.254:44695)
0926 20:19:53.406933 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:53.406958 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:53.407009 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:19:53.407875 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:53.407900 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:53.408429 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:53.408824 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:53.409334 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.410718 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.411674 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.412373 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.412535 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:53.413534 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:53.413541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:53.413558 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:53.414082 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:53.414156 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:53.414185 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:53.414295 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:53.414471 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 20:19:53.414832 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":116,"threads_started":1}
I20260926 20:19:53.416488 18475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559328ed5ecf409798f0b5acf79058b4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44695 } }
I20260926 20:19:53.416766 18475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:53.416862 18475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559328ed5ecf409798f0b5acf79058b4, State: Initialized, Role: FOLLOWER
I20260926 20:19:53.417433 18475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559328ed5ecf409798f0b5acf79058b4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44695 } }
I20260926 20:19:53.417598 18475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:19:53.417668 18475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:19:53.417780 18475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:53.418900 18475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559328ed5ecf409798f0b5acf79058b4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44695 } }
I20260926 20:19:53.419258 18475 leader_election.cc:302] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 559328ed5ecf409798f0b5acf79058b4; no voters:
I20260926 20:19:53.419633 18475 leader_election.cc:288] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:19:53.419757 18480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:53.420048 18480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [term 1 LEADER]: Becoming Leader. State: Replica: 559328ed5ecf409798f0b5acf79058b4, State: Running, Role: LEADER
I20260926 20:19:53.420383 18480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559328ed5ecf409798f0b5acf79058b4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44695 } }
I20260926 20:19:53.421051 18475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:19:53.421516 18481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 559328ed5ecf409798f0b5acf79058b4. Latest consensus state: current_term: 1 leader_uuid: "559328ed5ecf409798f0b5acf79058b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559328ed5ecf409798f0b5acf79058b4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44695 } } }
I20260926 20:19:53.421645 18481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:53.421962 18482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "559328ed5ecf409798f0b5acf79058b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559328ed5ecf409798f0b5acf79058b4" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 44695 } } }
I20260926 20:19:53.422068 18482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:53.422662 18484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:19:53.426079 18484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:19:53.432511 18484 catalog_manager.cc:1347] Generated new cluster ID: fd765d91d75c4a6699836ad1f3308237
I20260926 20:19:53.432626 18484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:19:53.448030 18484 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:19:53.449065 18484 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:19:53.461812 18484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4: Generated new TSK 0
I20260926 20:19:53.462818 18484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:19:53.473716 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44695
--builtin_ntp_servers=127.31.119.212:44845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:19:53.591127 18499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:53.591495 18499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:53.591581 18499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:53.595855 18499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:53.595986 18499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:53.596094 18499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:19:53.601192 18499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:44845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:53.602490 18499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:53.604022 18499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:53.615053 18504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:53.615003 18507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:53.615003 18505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:53.615244 18499 server_base.cc:1034] running on GCE node
I20260926 20:19:53.615819 18499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:53.616562 18499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:53.617827 18499 hybrid_clock.cc:648] HybridClock initialized: now 1790453993617830 us; error 59 us; skew 500 ppm
I20260926 20:19:53.629550 18499 webserver.cc:460] Webserver started at http://127.31.119.193:38925/ using document root <none> and password file <none>
I20260926 20:19:53.630257 18499 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:53.630376 18499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:53.630640 18499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:53.632920 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d"
format_stamp: "Formatted at 2026-09-26 20:19:53 on dist-test-slave-vnbh"
server_key: "5d47e907afa330d47a75fb8bf5367e0b"
server_key_iv: "ada9f62500e7b58e9a9608b65f49e095"
server_key_version: "encryptionkey@0"
I20260926 20:19:53.633574 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d"
format_stamp: "Formatted at 2026-09-26 20:19:53 on dist-test-slave-vnbh"
server_key: "5d47e907afa330d47a75fb8bf5367e0b"
server_key_iv: "ada9f62500e7b58e9a9608b65f49e095"
server_key_version: "encryptionkey@0"
I20260926 20:19:53.637573 18499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:19:53.640552 18513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:53.641983 18499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:53.642172 18499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d"
format_stamp: "Formatted at 2026-09-26 20:19:53 on dist-test-slave-vnbh"
server_key: "5d47e907afa330d47a75fb8bf5367e0b"
server_key_iv: "ada9f62500e7b58e9a9608b65f49e095"
server_key_version: "encryptionkey@0"
I20260926 20:19:53.642316 18499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:53.654191 18499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:53.655014 18499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:53.655253 18499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:53.655913 18499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:53.656931 18499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:53.657016 18499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:53.657088 18499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:53.657146 18499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:53.667652 18499 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:35967
I20260926 20:19:53.667668 18626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:35967 every 8 connection(s)
I20260926 20:19:53.668666 18499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:19:53.671175 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 18499
I20260926 20:19:53.671319 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:19:53.671600 32223 external_mini_cluster.cc:1428] Setting key 776dc32d85891afe505fd1a1df1c5421
I20260926 20:19:53.674063 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44695
--builtin_ntp_servers=127.31.119.212:44845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:19:53.681124 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.670784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:57443 (local address 127.31.119.254:44695)
0926 20:19:53.670958 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:53.670966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:53.672372 (+ 1406us) server_negotiation.cc:407] Connection header received
0926 20:19:53.674105 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:53.674109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:53.674175 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:53.674271 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:53.675869 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.676778 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.678194 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.678524 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.678632 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:53.680244 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:53.680247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:53.680249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:53.680266 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:53.680290 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:53.680298 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:53.680454 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:53.680830 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 20:19:53.680947 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 20:19:53.682768 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.671274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:44695 (local address 127.31.119.193:57443)
0926 20:19:53.672171 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:53.672233 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:53.673854 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:53.674647 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:53.674660 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:53.674794 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:53.675635 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:53.675664 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.676936 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.676939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:53.678033 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:53.678044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.678741 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.678748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:53.679025 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:53.679815 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:19:53.679818 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:53.679901 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:19:53.680014 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:53.680481 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:53.680494 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:53.680505 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 20:19:53.680681 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 20:19:53.681141 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":167,"threads_started":1}
I20260926 20:19:53.684517 18627 heartbeater.cc:347] Connected to a master server at 127.31.119.254:44695
I20260926 20:19:53.684836 18627 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:53.685571 18627 heartbeater.cc:511] Master 127.31.119.254:44695 requested a full tablet report, sending...
I20260926 20:19:53.687345 18439 ts_manager.cc:194] Registered new tserver with Master: 0ccf7413f5c94cc6b3213ea81e6bed2d (127.31.119.193:35967)
I20260926 20:19:53.688575 18439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.193:57443
W20260926 20:19:53.798084 18630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:53.798406 18630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:53.798450 18630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:53.803333 18630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:53.803403 18630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:53.803469 18630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:19:53.808841 18630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:44845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:53.810114 18630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:53.811601 18630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:53.821851 18636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:53.821859 18635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:53.822003 18638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:53.822373 18630 server_base.cc:1034] running on GCE node
I20260926 20:19:53.822882 18630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:53.823583 18630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:53.824754 18630 hybrid_clock.cc:648] HybridClock initialized: now 1790453993824733 us; error 33 us; skew 500 ppm
I20260926 20:19:53.826872 18630 webserver.cc:460] Webserver started at http://127.31.119.194:46629/ using document root <none> and password file <none>
I20260926 20:19:53.827477 18630 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:53.827541 18630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:53.827759 18630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:53.829996 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "da6837bb5e71471ea1fb68f58bed7c1a"
format_stamp: "Formatted at 2026-09-26 20:19:53 on dist-test-slave-vnbh"
server_key: "f011a4a9937dacd256061112ed29d863"
server_key_iv: "8335cde760162659e4146e5be1286f58"
server_key_version: "encryptionkey@0"
I20260926 20:19:53.830570 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "da6837bb5e71471ea1fb68f58bed7c1a"
format_stamp: "Formatted at 2026-09-26 20:19:53 on dist-test-slave-vnbh"
server_key: "f011a4a9937dacd256061112ed29d863"
server_key_iv: "8335cde760162659e4146e5be1286f58"
server_key_version: "encryptionkey@0"
I20260926 20:19:53.834443 18630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:19:53.836928 18644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:53.838251 18630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:19:53.838395 18630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "da6837bb5e71471ea1fb68f58bed7c1a"
format_stamp: "Formatted at 2026-09-26 20:19:53 on dist-test-slave-vnbh"
server_key: "f011a4a9937dacd256061112ed29d863"
server_key_iv: "8335cde760162659e4146e5be1286f58"
server_key_version: "encryptionkey@0"
I20260926 20:19:53.838518 18630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:53.858773 18630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:53.859624 18630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:53.859856 18630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:53.860499 18630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:53.861575 18630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:53.861651 18630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:53.861734 18630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:53.861768 18630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:53.871492 18630 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:34983
I20260926 20:19:53.871570 18757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:34983 every 8 connection(s)
I20260926 20:19:53.872557 18630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:19:53.873693 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 18630
I20260926 20:19:53.873818 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:19:53.874126 32223 external_mini_cluster.cc:1428] Setting key da3b8e83b95786f87c2c3b38c703f249
I20260926 20:19:53.876583 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44695
--builtin_ntp_servers=127.31.119.212:44845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:19:53.883963 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.875032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:37825 (local address 127.31.119.254:44695)
0926 20:19:53.875165 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:53.875168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:53.876674 (+ 1506us) server_negotiation.cc:407] Connection header received
0926 20:19:53.878356 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:53.878363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:53.878433 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:53.878536 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:53.879671 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.880243 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.881685 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.882001 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.882139 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:53.883408 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:53.883412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:53.883415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:53.883430 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:53.883456 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:53.883465 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:53.883551 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:53.883737 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 20:19:53.883805 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:19:53.884965 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:53.875689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:44695 (local address 127.31.119.194:37825)
0926 20:19:53.876514 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:53.876573 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:53.878095 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:53.878813 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:53.878826 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:53.878921 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:53.879483 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:53.879499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.880414 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.880418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:53.881546 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:53.881558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:53.882189 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:53.882194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:53.882380 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:53.883093 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:19:53.883095 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:53.883141 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:19:53.883213 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:53.883565 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:53.883576 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:53.883584 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:19:53.883726 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 20:19:53.884107 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":188,"threads_started":1}
I20260926 20:19:53.886296 18758 heartbeater.cc:347] Connected to a master server at 127.31.119.254:44695
I20260926 20:19:53.886570 18758 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:53.887125 18758 heartbeater.cc:511] Master 127.31.119.254:44695 requested a full tablet report, sending...
I20260926 20:19:53.888353 18439 ts_manager.cc:194] Registered new tserver with Master: da6837bb5e71471ea1fb68f58bed7c1a (127.31.119.194:34983)
I20260926 20:19:53.889120 18439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.194:37825
W20260926 20:19:53.994160 18761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:53.994450 18761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:53.994489 18761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:53.998615 18761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:53.998687 18761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:53.998750 18761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:19:54.003729 18761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:44845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:54.004889 18761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:54.006325 18761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:54.016896 18767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:54.017036 18761 server_base.cc:1034] running on GCE node
W20260926 20:19:54.016979 18769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:54.016899 18766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:54.017802 18761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:54.018536 18761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:54.019707 18761 hybrid_clock.cc:648] HybridClock initialized: now 1790453994019691 us; error 29 us; skew 500 ppm
I20260926 20:19:54.022076 18761 webserver.cc:460] Webserver started at http://127.31.119.195:37539/ using document root <none> and password file <none>
I20260926 20:19:54.022786 18761 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:54.022926 18761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:54.023144 18761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:54.025758 18761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "2b8b248438f54aa98cd73250b3f53e31"
format_stamp: "Formatted at 2026-09-26 20:19:54 on dist-test-slave-vnbh"
server_key: "3be01e1e7a44affe0bf4ecd40a754cb1"
server_key_iv: "01f96f81530304ea40efb35d8db1c626"
server_key_version: "encryptionkey@0"
I20260926 20:19:54.026577 18761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "2b8b248438f54aa98cd73250b3f53e31"
format_stamp: "Formatted at 2026-09-26 20:19:54 on dist-test-slave-vnbh"
server_key: "3be01e1e7a44affe0bf4ecd40a754cb1"
server_key_iv: "01f96f81530304ea40efb35d8db1c626"
server_key_version: "encryptionkey@0"
I20260926 20:19:54.030706 18761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:19:54.033699 18775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:54.035140 18761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:54.035254 18761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "2b8b248438f54aa98cd73250b3f53e31"
format_stamp: "Formatted at 2026-09-26 20:19:54 on dist-test-slave-vnbh"
server_key: "3be01e1e7a44affe0bf4ecd40a754cb1"
server_key_iv: "01f96f81530304ea40efb35d8db1c626"
server_key_version: "encryptionkey@0"
I20260926 20:19:54.035362 18761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:54.047219 18761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:54.048015 18761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:54.048221 18761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:54.048825 18761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:54.049825 18761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:54.049904 18761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:54.049969 18761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:54.050022 18761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:54.059029 18761 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:34787
I20260926 20:19:54.059048 18888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:34787 every 8 connection(s)
I20260926 20:19:54.060003 18761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:19:54.063791 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 18761
I20260926 20:19:54.063907 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:19:54.064199 32223 external_mini_cluster.cc:1428] Setting key 11ca3434506e85d421dec6fe205f669b
I20260926 20:19:54.070451 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.061975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:59333 (local address 127.31.119.254:44695)
0926 20:19:54.062143 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:54.062147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:54.063289 (+ 1142us) server_negotiation.cc:407] Connection header received
0926 20:19:54.065167 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:54.065170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:54.065229 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:54.065339 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:54.066413 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.067045 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.068287 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.068515 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.068637 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.069923 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:54.069927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:54.069929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:54.069945 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:54.069971 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:54.069980 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:54.070063 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:54.070228 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 20:19:54.070292 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:19:54.071430 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.062369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:44695 (local address 127.31.119.195:59333)
0926 20:19:54.063104 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:54.063147 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:54.064910 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:54.065546 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:54.065556 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:54.065676 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:54.066256 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.066273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.067188 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.067192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.068107 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.068116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.068681 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.068685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.068871 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.069620 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:19:54.069622 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:54.069667 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:19:54.069736 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:54.070065 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:54.070078 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:54.070085 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:19:54.070229 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 20:19:54.070566 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260926 20:19:54.072719 18889 heartbeater.cc:347] Connected to a master server at 127.31.119.254:44695
I20260926 20:19:54.072948 18889 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:54.073561 18889 heartbeater.cc:511] Master 127.31.119.254:44695 requested a full tablet report, sending...
I20260926 20:19:54.074647 18439 ts_manager.cc:194] Registered new tserver with Master: 2b8b248438f54aa98cd73250b3f53e31 (127.31.119.195:34787)
I20260926 20:19:54.075374 18439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.195:59333
I20260926 20:19:54.078744 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:19:54.084959 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.080850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.31.119.254:44695)
0926 20:19:54.080993 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:54.081001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:54.081160 (+ 159us) server_negotiation.cc:407] Connection header received
0926 20:19:54.081249 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:54.081251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:54.081354 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:54.081434 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:54.081790 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.082291 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.083184 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.083399 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.083488 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.084390 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:54.084393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:54.084395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:54.084407 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:54.084426 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:54.084434 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:54.084588 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:54.084804 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 20:19:54.084839 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:19:54.088127 18439 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:19:54.091231 18439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:19:54.107923 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.102003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35967 (local address 127.0.0.1:34850)
0926 20:19:54.103097 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:54.103111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:54.103228 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:54.103665 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:54.103669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:54.103687 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:54.103792 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.103803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.105096 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.105103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.106414 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.106422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.107503 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.107507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.107592 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.107616 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:19:54.107678 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:19:54.107744 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":4,"thread_start_us":108,"threads_started":1}
I20260926 20:19:54.108036 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.102461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.31.119.193:35967)
0926 20:19:54.102921 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:54.102937 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:54.103241 (+ 304us) server_negotiation.cc:407] Connection header received
0926 20:19:54.103352 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:54.103358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:54.103505 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:54.103677 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:54.103993 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.104903 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.106549 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.107362 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.107493 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.107675 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:54.107729 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:19:54.107855 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":106,"threads_started":1}
I20260926 20:19:54.108181 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.101171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.31.119.194:34983)
0926 20:19:54.101646 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:54.101660 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:54.102150 (+ 490us) server_negotiation.cc:407] Connection header received
0926 20:19:54.102239 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:54.102247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:54.102481 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:54.102635 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:54.103269 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.104135 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.106236 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.107067 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.107215 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.107425 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:54.107830 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 20:19:54.107996 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":151,"threads_started":1}
I20260926 20:19:54.108404 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.100984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:34983 (local address 127.0.0.1:35368)
0926 20:19:54.101665 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:54.101725 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:54.101951 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:54.102831 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:54.102838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:54.102862 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:54.103099 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.103106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.104284 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.104288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.106069 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.106080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.107212 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.107216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.107360 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.107423 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 20:19:54.107539 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:19:54.107679 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":76,"thread_start_us":121,"threads_started":1}
I20260926 20:19:54.109459 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.101257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:34787 (local address 127.0.0.1:37944)
0926 20:19:54.102090 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:54.102109 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:54.102207 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:54.103004 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:54.103007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:54.103023 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:54.103102 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.103106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.105831 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.105834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.107063 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.107072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.109015 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.109019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.109114 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.109136 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:19:54.109214 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:19:54.109315 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":26,"thread_start_us":111,"threads_started":1}
I20260926 20:19:54.109915 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.101925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.31.119.195:34787)
0926 20:19:54.102434 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:54.102445 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:54.102467 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:19:54.102539 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:54.102547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:54.102723 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:54.102866 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:54.103861 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.105632 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.107229 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.108470 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.108769 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.108935 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:54.109705 (+ 770us) server_negotiation.cc:299] Negotiation successful
0926 20:19:54.109795 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":168,"threads_started":1}
I20260926 20:19:54.111528 18692 tablet_service.cc:1467] Processing CreateTablet for tablet 8478f5b7371f47f4a3b8f8f28a33fec9 (DEFAULT_TABLE table=test-table [id=dc70a29409694450b98451923dac4cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:54.111524 18561 tablet_service.cc:1467] Processing CreateTablet for tablet 8478f5b7371f47f4a3b8f8f28a33fec9 (DEFAULT_TABLE table=test-table [id=dc70a29409694450b98451923dac4cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:54.112344 18823 tablet_service.cc:1467] Processing CreateTablet for tablet 8478f5b7371f47f4a3b8f8f28a33fec9 (DEFAULT_TABLE table=test-table [id=dc70a29409694450b98451923dac4cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:54.112761 18561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8478f5b7371f47f4a3b8f8f28a33fec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:54.112761 18692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8478f5b7371f47f4a3b8f8f28a33fec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:54.113332 18823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8478f5b7371f47f4a3b8f8f28a33fec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:54.119625 18906 tablet_bootstrap.cc:492] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: Bootstrap starting.
I20260926 20:19:54.121840 18908 tablet_bootstrap.cc:492] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: Bootstrap starting.
I20260926 20:19:54.121840 18907 tablet_bootstrap.cc:492] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d: Bootstrap starting.
I20260926 20:19:54.122655 18906 tablet_bootstrap.cc:654] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:54.123935 18906 log.cc:824] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:54.124286 18907 tablet_bootstrap.cc:654] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:54.124286 18908 tablet_bootstrap.cc:654] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:54.125231 18908 log.cc:824] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:54.125231 18907 log.cc:824] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:54.126828 18906 tablet_bootstrap.cc:492] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: No bootstrap required, opened a new log
I20260926 20:19:54.127065 18906 ts_tablet_manager.cc:1397] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260926 20:19:54.127488 18908 tablet_bootstrap.cc:492] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: No bootstrap required, opened a new log
I20260926 20:19:54.127699 18908 ts_tablet_manager.cc:1397] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:19:54.127733 18907 tablet_bootstrap.cc:492] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d: No bootstrap required, opened a new log
I20260926 20:19:54.127972 18907 ts_tablet_manager.cc:1397] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260926 20:19:54.131209 18908 raft_consensus.cc:348] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34983 } } peers { permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 } } peers { permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 } }
I20260926 20:19:54.131544 18908 raft_consensus.cc:374] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:54.131601 18908 raft_consensus.cc:729] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6837bb5e71471ea1fb68f58bed7c1a, State: Initialized, Role: FOLLOWER
I20260926 20:19:54.131677 18906 raft_consensus.cc:348] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34983 } } peers { permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 } } peers { permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 } }
I20260926 20:19:54.131948 18906 raft_consensus.cc:374] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:54.131999 18906 raft_consensus.cc:729] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b8b248438f54aa98cd73250b3f53e31, State: Initialized, Role: FOLLOWER
I20260926 20:19:54.132095 18908 consensus_queue.cc:260] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34983 } } peers { permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 } } peers { permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 } }
I20260926 20:19:54.132279 18907 raft_consensus.cc:348] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34983 } } peers { permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 } } peers { permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 } }
I20260926 20:19:54.132519 18907 raft_consensus.cc:374] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:54.132566 18907 raft_consensus.cc:729] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ccf7413f5c94cc6b3213ea81e6bed2d, State: Initialized, Role: FOLLOWER
I20260926 20:19:54.132496 18906 consensus_queue.cc:260] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34983 } } peers { permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 } } peers { permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 } }
I20260926 20:19:54.132980 18907 consensus_queue.cc:260] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34983 } } peers { permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 } } peers { permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 } }
I20260926 20:19:54.133100 18758 heartbeater.cc:503] Master 127.31.119.254:44695 was elected leader, sending a full tablet report...
I20260926 20:19:54.133261 18908 ts_tablet_manager.cc:1428] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:19:54.133610 18906 ts_tablet_manager.cc:1428] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260926 20:19:54.133925 18907 ts_tablet_manager.cc:1428] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260926 20:19:54.133945 18889 heartbeater.cc:503] Master 127.31.119.254:44695 was elected leader, sending a full tablet report...
I20260926 20:19:54.134351 18627 heartbeater.cc:503] Master 127.31.119.254:44695 was elected leader, sending a full tablet report...
W20260926 20:19:54.169979 18628 tablet.cc:2369] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:19:54.198844 18912 raft_consensus.cc:482] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:19:54.199079 18912 raft_consensus.cc:504] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34983 } } peers { permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 } } peers { permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 } }
I20260926 20:19:54.200518 18912 leader_election.cc:288] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b8b248438f54aa98cd73250b3f53e31 (127.31.119.195:34787), 0ccf7413f5c94cc6b3213ea81e6bed2d (127.31.119.193:35967)
I20260926 20:19:54.205186 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.200785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:34787 (local address 127.31.119.194:34797)
0926 20:19:54.200986 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:54.201005 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:54.201130 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:54.201486 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:54.201489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:54.201507 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:54.201643 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.201651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.202668 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.202672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.203814 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.203824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.204680 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.204684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.204798 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.204827 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 20:19:54.204921 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:19:54.205007 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":37}
I20260926 20:19:54.205427 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.200940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:34797 (local address 127.31.119.195:34787)
0926 20:19:54.201086 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:54.201090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:54.201106 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:19:54.201278 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:54.201281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:54.201366 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:54.201757 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:54.201870 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.202537 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.203974 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.204514 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.204623 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.204680 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:54.204984 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 20:19:54.205060 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:19:54.205896 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.201104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35967 (local address 127.31.119.194:46999)
0926 20:19:54.201598 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:54.201612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:54.201761 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:54.202060 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:54.202063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:54.202080 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:54.202212 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.202218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.203242 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.203245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.204208 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:54.204219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.205253 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.205257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:54.205417 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.205445 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:19:54.205548 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:19:54.205638 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260926 20:19:54.205955 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.201405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:46999 (local address 127.31.119.193:35967)
0926 20:19:54.201559 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:54.201564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:54.201719 (+ 155us) server_negotiation.cc:407] Connection header received
0926 20:19:54.201867 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:54.201871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:54.201929 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:54.202032 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:54.202325 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.203077 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.204377 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.205067 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.205189 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.205254 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:54.205695 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 20:19:54.205822 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:19:54.206164 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8478f5b7371f47f4a3b8f8f28a33fec9" candidate_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b8b248438f54aa98cd73250b3f53e31" is_pre_election: true
I20260926 20:19:54.206595 18843 raft_consensus.cc:2454] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da6837bb5e71471ea1fb68f58bed7c1a in term 0.
I20260926 20:19:54.206743 18581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8478f5b7371f47f4a3b8f8f28a33fec9" candidate_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" is_pre_election: true
I20260926 20:19:54.207046 18581 raft_consensus.cc:2454] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da6837bb5e71471ea1fb68f58bed7c1a in term 0.
I20260926 20:19:54.207212 18646 leader_election.cc:302] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b8b248438f54aa98cd73250b3f53e31, da6837bb5e71471ea1fb68f58bed7c1a; no voters:
I20260926 20:19:54.207535 18912 raft_consensus.cc:2781] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:19:54.207652 18912 raft_consensus.cc:482] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:19:54.207721 18912 raft_consensus.cc:3037] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:54.208967 18912 raft_consensus.cc:504] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34983 } } peers { permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 } } peers { permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 } }
I20260926 20:19:54.209467 18912 leader_election.cc:288] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [CANDIDATE]: Term 1 election: Requested vote from peers 2b8b248438f54aa98cd73250b3f53e31 (127.31.119.195:34787), 0ccf7413f5c94cc6b3213ea81e6bed2d (127.31.119.193:35967)
I20260926 20:19:54.209930 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8478f5b7371f47f4a3b8f8f28a33fec9" candidate_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b8b248438f54aa98cd73250b3f53e31"
I20260926 20:19:54.210000 18581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8478f5b7371f47f4a3b8f8f28a33fec9" candidate_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d"
I20260926 20:19:54.210063 18843 raft_consensus.cc:3037] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:54.210206 18581 raft_consensus.cc:3037] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:54.211287 18843 raft_consensus.cc:2454] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6837bb5e71471ea1fb68f58bed7c1a in term 1.
I20260926 20:19:54.211387 18581 raft_consensus.cc:2454] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6837bb5e71471ea1fb68f58bed7c1a in term 1.
I20260926 20:19:54.211674 18646 leader_election.cc:302] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b8b248438f54aa98cd73250b3f53e31, da6837bb5e71471ea1fb68f58bed7c1a; no voters:
I20260926 20:19:54.211856 18912 raft_consensus.cc:2781] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:54.212097 18912 raft_consensus.cc:686] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 1 LEADER]: Becoming Leader. State: Replica: da6837bb5e71471ea1fb68f58bed7c1a, State: Running, Role: LEADER
I20260926 20:19:54.212512 18912 consensus_queue.cc:237] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34983 } } peers { permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 } } peers { permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 } }
I20260926 20:19:54.216120 18438 catalog_manager.cc:5495] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a reported cstate change: term changed from 0 to 1, leader changed from <none> to da6837bb5e71471ea1fb68f58bed7c1a (127.31.119.194). New cstate: current_term: 1 leader_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da6837bb5e71471ea1fb68f58bed7c1a" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 34983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 } health_report { overall_health: UNKNOWN } } }
I20260926 20:19:54.236267 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.231973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.31.119.194:34983)
0926 20:19:54.232123 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:54.232127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:54.232141 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:19:54.232215 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:54.232218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:54.232297 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:54.232394 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:19:54.232624 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.233155 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.234419 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:54.234639 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:54.234729 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:54.235165 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:19:54.235602 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:19:54.235733 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:54.236001 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 20:19:54.236056 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:19:54.243073 18843 raft_consensus.cc:1261] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31 [term 1 FOLLOWER]: Refusing update from remote peer da6837bb5e71471ea1fb68f58bed7c1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:54.243068 18581 raft_consensus.cc:1261] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d [term 1 FOLLOWER]: Refusing update from remote peer da6837bb5e71471ea1fb68f58bed7c1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:54.243871 18912 consensus_queue.cc:1035] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b8b248438f54aa98cd73250b3f53e31" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:54.244103 18916 consensus_queue.cc:1035] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ccf7413f5c94cc6b3213ea81e6bed2d" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:54.253170 18921 mvcc.cc:205] Tried to move back new op lower bound from 7333699560410775552 to 7333699560297652224. Current Snapshot: MvccSnapshot[applied={T|T < 7333699560410775552}]
I20260926 20:19:54.253430 18920 mvcc.cc:205] Tried to move back new op lower bound from 7333699560410775552 to 7333699560297652224. Current Snapshot: MvccSnapshot[applied={T|T < 7333699560410775552}]
I20260926 20:19:54.262315 18439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:54.262530 18439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:54.265370 18439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 559328ed5ecf409798f0b5acf79058b4: Sending DeleteTablet for 3 replicas of tablet 8478f5b7371f47f4a3b8f8f28a33fec9
I20260926 20:19:54.266304 18692 tablet_service.cc:1514] Processing DeleteTablet for tablet 8478f5b7371f47f4a3b8f8f28a33fec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:54 UTC) from {username='slave'} at 127.0.0.1:35368
I20260926 20:19:54.266367 18823 tablet_service.cc:1514] Processing DeleteTablet for tablet 8478f5b7371f47f4a3b8f8f28a33fec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:54 UTC) from {username='slave'} at 127.0.0.1:37944
I20260926 20:19:54.266727 18561 tablet_service.cc:1514] Processing DeleteTablet for tablet 8478f5b7371f47f4a3b8f8f28a33fec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:54 UTC) from {username='slave'} at 127.0.0.1:34850
I20260926 20:19:54.267254 18931 tablet_replica.cc:330] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 0ccf7413f5c94cc6b3213ea81e6bed2d: stopping tablet replica
I20260926 20:19:54.267315 18929 tablet_replica.cc:330] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: stopping tablet replica
I20260926 20:19:54.267379 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 18499
I20260926 20:19:54.267659 18929 raft_consensus.cc:2229] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:54.268019 18929 raft_consensus.cc:2258] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:54.268419 18930 tablet_replica.cc:330] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: stopping tablet replica
I20260926 20:19:54.268752 18930 raft_consensus.cc:2229] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:19:54.269127 18930 raft_consensus.cc:2258] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:54.270606 18929 ts_tablet_manager.cc:1905] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:54.271576 18930 ts_tablet_manager.cc:1905] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:54.273524 18929 ts_tablet_manager.cc:1918] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:54.273619 18929 log.cc:1196] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/8478f5b7371f47f4a3b8f8f28a33fec9
I20260926 20:19:54.273922 18929 ts_tablet_manager.cc:1939] T 8478f5b7371f47f4a3b8f8f28a33fec9 P 2b8b248438f54aa98cd73250b3f53e31: Deleting consensus metadata
I20260926 20:19:54.275079 18930 ts_tablet_manager.cc:1918] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:54.275192 18930 log.cc:1196] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/8478f5b7371f47f4a3b8f8f28a33fec9
I20260926 20:19:54.275529 18930 ts_tablet_manager.cc:1939] T 8478f5b7371f47f4a3b8f8f28a33fec9 P da6837bb5e71471ea1fb68f58bed7c1a: Deleting consensus metadata
I20260926 20:19:54.275724 18424 catalog_manager.cc:4841] TS 2b8b248438f54aa98cd73250b3f53e31 (127.31.119.195:34787): tablet 8478f5b7371f47f4a3b8f8f28a33fec9 (table test-table [id=dc70a29409694450b98451923dac4cc9]) successfully deleted
W20260926 20:19:54.276271 18424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:35967 (error 108)
I20260926 20:19:54.276515 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 18630
I20260926 20:19:54.278425 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.278125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35967 (local address 127.0.0.1:34860)
0926 20:19:54.278265 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:54.278350 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:35967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 20:19:54.278589 18424 catalog_manager.cc:4584] TS 0ccf7413f5c94cc6b3213ea81e6bed2d (127.31.119.193:35967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8478f5b7371f47f4a3b8f8f28a33fec9: Network error: Client connection negotiation failed: client connection to 127.31.119.193:35967: connect: Connection refused (error 111)
I20260926 20:19:54.282817 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 18761
I20260926 20:19:54.283892 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:54.282971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:34983 (local address 127.0.0.1:35394)
0926 20:19:54.283721 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:54.283790 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.194:34983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642}
I20260926 20:19:54.289165 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 18408
2026-09-26T20:19:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1103 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:19:54 dist-test-slave-vnbh krb5kdc[18935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:19:54 dist-test-slave-vnbh krb5kdc[18935](info): set up 2 sockets
Sep 26 20:19:54 dist-test-slave-vnbh krb5kdc[18935](info): commencing operation
krb5kdc: starting...
W20260926 20:19:55.354499 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:19:55 dist-test-slave-vnbh krb5kdc[18935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:19:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:19:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:19:55.485862 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38283
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:55.601817 18948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:55.602151 18948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:55.602200 18948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:55.606495 18948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:19:55.606567 18948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:55.606587 18948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:55.606604 18948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:19:55.606619 18948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:19:55.612488 18948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:38283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:55.613658 18948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:55.614686 18948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:55.624081 18954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:55.624120 18953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:55.624176 18948 server_base.cc:1034] running on GCE node
W20260926 20:19:55.624081 18956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:55.625247 18948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:55.626493 18948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:55.627698 18948 hybrid_clock.cc:648] HybridClock initialized: now 1790453995627664 us; error 48 us; skew 500 ppm
Sep 26 20:19:55 dist-test-slave-vnbh krb5kdc[18935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453995, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:55.630725 18948 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:19:55.631788 18948 webserver.cc:460] Webserver started at http://127.31.119.254:38743/ using document root <none> and password file <none>
I20260926 20:19:55.632333 18948 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:55.632400 18948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:55.632571 18948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:55.634795 18948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "8907cd9576f34e888262d7d0012ee02e"
format_stamp: "Formatted at 2026-09-26 20:19:55 on dist-test-slave-vnbh"
server_key: "354484bc0c0f893696f50c7e730401bb"
server_key_iv: "907d3d176ffa44f8d52a6f8db45f215b"
server_key_version: "encryptionkey@0"
I20260926 20:19:55.635324 18948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "8907cd9576f34e888262d7d0012ee02e"
format_stamp: "Formatted at 2026-09-26 20:19:55 on dist-test-slave-vnbh"
server_key: "354484bc0c0f893696f50c7e730401bb"
server_key_iv: "907d3d176ffa44f8d52a6f8db45f215b"
server_key_version: "encryptionkey@0"
I20260926 20:19:55.639528 18948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:19:55.642303 18963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:55.643803 18948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:55.643921 18948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "8907cd9576f34e888262d7d0012ee02e"
format_stamp: "Formatted at 2026-09-26 20:19:55 on dist-test-slave-vnbh"
server_key: "354484bc0c0f893696f50c7e730401bb"
server_key_iv: "907d3d176ffa44f8d52a6f8db45f215b"
server_key_version: "encryptionkey@0"
I20260926 20:19:55.644119 18948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:55.679316 18948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:55.682997 18948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:55.683218 18948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:55.690862 18948 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:38283
I20260926 20:19:55.690879 19015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:38283 every 8 connection(s)
I20260926 20:19:55.691888 18948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:19:55.693275 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 18948
I20260926 20:19:55.693481 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:19:55.693758 32223 external_mini_cluster.cc:1428] Setting key 1f6eae962625a31cbcdf2654592e2b91
I20260926 20:19:55.695330 19016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:55.702307 19016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e: Bootstrap starting.
Sep 26 20:19:55 dist-test-slave-vnbh krb5kdc[18935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:55.704919 19016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:55.705950 19016 log.cc:824] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:55.708396 19016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e: No bootstrap required, opened a new log
I20260926 20:19:55.709470 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:55.695054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.31.119.254:38283)
0926 20:19:55.695490 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:55.695510 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:55.695551 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:19:55.696383 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:55.696404 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:55.696845 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:55.697208 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:55.697996 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:55.699509 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:55.700385 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:55.701107 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:55.701318 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:55.704003 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:55.704009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:55.704025 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:55.704059 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:55.706241 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:55.706690 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:55.706697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:55.706705 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:55.706851 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:55.707202 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:55.707204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:55.707205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:55.707599 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:55.707751 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:55.708112 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 20:19:55.708468 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
I20260926 20:19:55.712052 19016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907cd9576f34e888262d7d0012ee02e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38283 } }
I20260926 20:19:55.712358 19016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:55.712450 19016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8907cd9576f34e888262d7d0012ee02e, State: Initialized, Role: FOLLOWER
I20260926 20:19:55.712908 19016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907cd9576f34e888262d7d0012ee02e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38283 } }
I20260926 20:19:55.713025 19016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:19:55.713114 19016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:19:55.713245 19016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:55.714214 19016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907cd9576f34e888262d7d0012ee02e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38283 } }
I20260926 20:19:55.714573 19016 leader_election.cc:302] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8907cd9576f34e888262d7d0012ee02e; no voters:
I20260926 20:19:55.714893 19016 leader_election.cc:288] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:19:55.715080 19021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:55.715329 19021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [term 1 LEADER]: Becoming Leader. State: Replica: 8907cd9576f34e888262d7d0012ee02e, State: Running, Role: LEADER
I20260926 20:19:55.715801 19021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907cd9576f34e888262d7d0012ee02e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38283 } }
I20260926 20:19:55.716194 19016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:19:55.717535 19023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8907cd9576f34e888262d7d0012ee02e. Latest consensus state: current_term: 1 leader_uuid: "8907cd9576f34e888262d7d0012ee02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907cd9576f34e888262d7d0012ee02e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38283 } } }
I20260926 20:19:55.717671 19023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:55.718102 19030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:19:55.718067 19022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8907cd9576f34e888262d7d0012ee02e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8907cd9576f34e888262d7d0012ee02e" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 38283 } } }
I20260926 20:19:55.718195 19022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:55.721146 19030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:19:55.726555 19030 catalog_manager.cc:1347] Generated new cluster ID: 58fae950375c4633adb65954ed1ff95f
I20260926 20:19:55.726626 19030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:19:55.738663 19030 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:19:55.739765 19030 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:19:55.749709 19030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e: Generated new TSK 0
I20260926 20:19:55.750572 19030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.119.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
Entry for principal kudu/127.31.119.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab.
I20260926 20:19:55.792086 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38283
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:55.910446 19042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:55.910758 19042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:55.910804 19042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:55.915124 19042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:55.915205 19042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:55.915270 19042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:19:55.921252 19042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.193.keytab
--principal=kudu/127.31.119.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:55.922399 19042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:55.923445 19042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:55.933799 19047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:55.933878 19048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:55.933912 19042 server_base.cc:1034] running on GCE node
W20260926 20:19:55.933804 19050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:55.934497 19042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:55.935319 19042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:55.936534 19042 hybrid_clock.cc:648] HybridClock initialized: now 1790453995936517 us; error 30 us; skew 500 ppm
Sep 26 20:19:55 dist-test-slave-vnbh krb5kdc[18935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453995, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:55.940091 19042 init.cc:377] Logged in from keytab as kudu/127.31.119.193@KRBTEST.COM (short username kudu)
I20260926 20:19:55.941118 19042 webserver.cc:460] Webserver started at http://127.31.119.193:43267/ using document root <none> and password file <none>
I20260926 20:19:55.941774 19042 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:55.941841 19042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:55.942068 19042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:55.944303 19042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "853077bf34734e4780be6b4b86062288"
format_stamp: "Formatted at 2026-09-26 20:19:55 on dist-test-slave-vnbh"
server_key: "9bbf0d6ff647279efd4e23b27c98c88a"
server_key_iv: "3d6ddf3262e28d1e8c75004bdf15e142"
server_key_version: "encryptionkey@0"
I20260926 20:19:55.944949 19042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "853077bf34734e4780be6b4b86062288"
format_stamp: "Formatted at 2026-09-26 20:19:55 on dist-test-slave-vnbh"
server_key: "9bbf0d6ff647279efd4e23b27c98c88a"
server_key_iv: "3d6ddf3262e28d1e8c75004bdf15e142"
server_key_version: "encryptionkey@0"
I20260926 20:19:55.949257 19042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:19:55.951965 19057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:55.953426 19042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:55.953573 19042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "853077bf34734e4780be6b4b86062288"
format_stamp: "Formatted at 2026-09-26 20:19:55 on dist-test-slave-vnbh"
server_key: "9bbf0d6ff647279efd4e23b27c98c88a"
server_key_iv: "3d6ddf3262e28d1e8c75004bdf15e142"
server_key_version: "encryptionkey@0"
I20260926 20:19:55.953696 19042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:55.969933 19042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:55.973397 19042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:55.973618 19042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:55.974289 19042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:55.975148 19042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:55.975234 19042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:55.975329 19042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:55.975381 19042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:55.984478 19042 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:39309
I20260926 20:19:55.984505 19170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:39309 every 8 connection(s)
I20260926 20:19:55.985427 19042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:19:55.989226 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 19042
I20260926 20:19:55.989388 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:19:55.989737 32223 external_mini_cluster.cc:1428] Setting key b1952745dc6d0db4d764099856b2e2a0
Sep 26 20:19:55 dist-test-slave-vnbh krb5kdc[18935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453995, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.193@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.119.194@KRBTEST.COM; defaulting to no policy
I20260926 20:19:56.001449 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:55.987393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:38203 (local address 127.31.119.254:38283)
0926 20:19:55.987559 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:55.987563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:55.988393 (+ 830us) server_negotiation.cc:407] Connection header received
0926 20:19:55.990237 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:55.990240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:55.990333 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:55.990428 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:55.992043 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:55.992605 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:55.994055 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:55.994301 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:55.994421 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:55.997398 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:55.997401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:55.997403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:55.997429 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:55.999018 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:55.999622 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:55.999625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:55.999626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:55.999702 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:56.000108 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:56.000111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:56.000113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:56.000356 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:56.000448 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.001074 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.001199 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:19:56.002421 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:55.987738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38283 (local address 127.31.119.193:38203)
0926 20:19:55.988221 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:55.988267 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:55.989923 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:55.990680 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:55.990688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:55.991138 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:55.991864 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:55.991889 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:55.992753 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:55.992759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:55.993918 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:55.993928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:55.994465 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:55.994469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:55.994654 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:55.995321 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:55.995350 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:55.997179 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:55.999148 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:55.999155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:55.999172 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:55.999447 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:55.999816 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:55.999819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:55.999820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:55.999970 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:56.000520 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:56.000532 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:56.000815 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 20:19:56.000985 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 20:19:56.001333 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.194@KRBTEST.COM" created.
I20260926 20:19:56.004066 19171 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38283
I20260926 20:19:56.004383 19171 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:56.005162 19171 heartbeater.cc:511] Master 127.31.119.254:38283 requested a full tablet report, sending...
I20260926 20:19:56.007054 18980 ts_manager.cc:194] Registered new tserver with Master: 853077bf34734e4780be6b4b86062288 (127.31.119.193:39309)
I20260926 20:19:56.008831 18980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.193@KRBTEST.COM'} at 127.31.119.193:38203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
Entry for principal kudu/127.31.119.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab.
I20260926 20:19:56.021584 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38283
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:56.134264 19176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:56.134529 19176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:56.134573 19176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:56.138725 19176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:56.138803 19176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:56.138868 19176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:19:56.144634 19176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.194.keytab
--principal=kudu/127.31.119.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:56.145933 19176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:56.146998 19176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:56.157648 19181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:56.157660 19184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:56.157629 19182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:56.158517 19176 server_base.cc:1034] running on GCE node
I20260926 20:19:56.159068 19176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:56.159801 19176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:56.161007 19176 hybrid_clock.cc:648] HybridClock initialized: now 1790453996160988 us; error 34 us; skew 500 ppm
Sep 26 20:19:56 dist-test-slave-vnbh krb5kdc[18935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453996, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:56.164407 19176 init.cc:377] Logged in from keytab as kudu/127.31.119.194@KRBTEST.COM (short username kudu)
I20260926 20:19:56.165581 19176 webserver.cc:460] Webserver started at http://127.31.119.194:35581/ using document root <none> and password file <none>
I20260926 20:19:56.166211 19176 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:56.166311 19176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:56.166548 19176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:56.168694 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "b5c9ddca231d48f1b332f56e87081077"
format_stamp: "Formatted at 2026-09-26 20:19:56 on dist-test-slave-vnbh"
server_key: "d697399ea748849734214e579ac54c53"
server_key_iv: "3b6415b5180fb54051370f78e2f62189"
server_key_version: "encryptionkey@0"
I20260926 20:19:56.169369 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "b5c9ddca231d48f1b332f56e87081077"
format_stamp: "Formatted at 2026-09-26 20:19:56 on dist-test-slave-vnbh"
server_key: "d697399ea748849734214e579ac54c53"
server_key_iv: "3b6415b5180fb54051370f78e2f62189"
server_key_version: "encryptionkey@0"
I20260926 20:19:56.173388 19176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:19:56.176280 19191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:56.177851 19176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 20:19:56.178005 19176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "b5c9ddca231d48f1b332f56e87081077"
format_stamp: "Formatted at 2026-09-26 20:19:56 on dist-test-slave-vnbh"
server_key: "d697399ea748849734214e579ac54c53"
server_key_iv: "3b6415b5180fb54051370f78e2f62189"
server_key_version: "encryptionkey@0"
I20260926 20:19:56.178133 19176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:56.201118 19176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:56.204905 19176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:56.205153 19176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:56.205858 19176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:56.206756 19176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:56.206835 19176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:56.206952 19176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:56.206986 19176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:56.216723 19176 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:37851
I20260926 20:19:56.216801 19304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:37851 every 8 connection(s)
I20260926 20:19:56.217952 19176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:19:56.218802 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 19176
I20260926 20:19:56.218911 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:19:56.219138 32223 external_mini_cluster.cc:1428] Setting key fcbd13b48d62aebd1e0b647db0ef6679
WARNING: no policy specified for kudu/127.31.119.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.195@KRBTEST.COM" created.
Sep 26 20:19:56 dist-test-slave-vnbh krb5kdc[18935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453996, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.194@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:56.236506 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.220829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:57457 (local address 127.31.119.254:38283)
0926 20:19:56.220961 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:56.220964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:56.221470 (+ 506us) server_negotiation.cc:407] Connection header received
0926 20:19:56.222838 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:56.222841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:56.222897 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:56.222974 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:56.224506 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.225064 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.226369 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.226606 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.226726 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.232334 (+ 5608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:56.232338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:56.232341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:56.232373 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:56.234174 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:56.234801 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:56.234806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:56.234808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:56.234895 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:56.235278 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:56.235281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:56.235283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:56.235488 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:56.235597 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.236204 (+ 607us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.236338 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:19:56.237504 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.220558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38283 (local address 127.31.119.194:57457)
0926 20:19:56.221272 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:56.221375 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:56.222617 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:56.223314 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:56.223330 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:56.223809 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:56.224331 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.224346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.225223 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.225226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.226226 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.226236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.226776 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.226781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.226960 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.227627 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:56.227645 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:56.232091 (+ 4446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:56.234334 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:56.234342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:56.234361 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:56.234668 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:56.234992 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:56.234996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:56.234998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:56.235169 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:56.235597 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:56.235607 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:56.235932 (+ 325us) client_negotiation.cc:769] Sending connection context
0926 20:19:56.236136 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 20:19:56.236539 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":135,"threads_started":1}
I20260926 20:19:56.238976 19305 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38283
I20260926 20:19:56.239257 19305 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:56.239799 19305 heartbeater.cc:511] Master 127.31.119.254:38283 requested a full tablet report, sending...
I20260926 20:19:56.241081 18980 ts_manager.cc:194] Registered new tserver with Master: b5c9ddca231d48f1b332f56e87081077 (127.31.119.194:37851)
I20260926 20:19:56.241782 18980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.194@KRBTEST.COM'} at 127.31.119.194:57457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
Entry for principal kudu/127.31.119.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab.
I20260926 20:19:56.249789 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38283
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:56.366154 19310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:56.366433 19310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:56.366479 19310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:56.370558 19310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:56.370630 19310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:56.370690 19310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:19:56.376341 19310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.195.keytab
--principal=kudu/127.31.119.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:38283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:56.377461 19310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:56.378469 19310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:19:56.389173 19310 server_base.cc:1034] running on GCE node
W20260926 20:19:56.389115 19315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:56.389094 19316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:56.389108 19318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:56.389899 19310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:56.390702 19310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:56.391928 19310 hybrid_clock.cc:648] HybridClock initialized: now 1790453996391909 us; error 32 us; skew 500 ppm
Sep 26 20:19:56 dist-test-slave-vnbh krb5kdc[18935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453996, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:56.395090 19310 init.cc:377] Logged in from keytab as kudu/127.31.119.195@KRBTEST.COM (short username kudu)
I20260926 20:19:56.395993 19310 webserver.cc:460] Webserver started at http://127.31.119.195:37735/ using document root <none> and password file <none>
I20260926 20:19:56.396603 19310 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:56.396667 19310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:56.396844 19310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:56.399016 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "64959d59003f4ca2b7bb4d0dae340364"
format_stamp: "Formatted at 2026-09-26 20:19:56 on dist-test-slave-vnbh"
server_key: "d5c62269431b1ee03cb1ad9ec47106ed"
server_key_iv: "2ef1fefb0a93ffaa485b7782cc442aed"
server_key_version: "encryptionkey@0"
I20260926 20:19:56.399559 19310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "64959d59003f4ca2b7bb4d0dae340364"
format_stamp: "Formatted at 2026-09-26 20:19:56 on dist-test-slave-vnbh"
server_key: "d5c62269431b1ee03cb1ad9ec47106ed"
server_key_iv: "2ef1fefb0a93ffaa485b7782cc442aed"
server_key_version: "encryptionkey@0"
I20260926 20:19:56.403635 19310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:19:56.406315 19325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:56.407624 19310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:56.407753 19310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "64959d59003f4ca2b7bb4d0dae340364"
format_stamp: "Formatted at 2026-09-26 20:19:56 on dist-test-slave-vnbh"
server_key: "d5c62269431b1ee03cb1ad9ec47106ed"
server_key_iv: "2ef1fefb0a93ffaa485b7782cc442aed"
server_key_version: "encryptionkey@0"
I20260926 20:19:56.407881 19310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:56.423118 19310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:56.426548 19310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:56.426792 19310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:56.427404 19310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:56.428252 19310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:56.428325 19310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:56.428426 19310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:56.428474 19310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:56.437775 19310 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:43823
I20260926 20:19:56.438141 19438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:43823 every 8 connection(s)
I20260926 20:19:56.438810 19310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:19:56.447352 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 19310
I20260926 20:19:56.447510 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:19:56.447803 32223 external_mini_cluster.cc:1428] Setting key ffec0843693134ca169b87b4ee5b2cc7
Sep 26 20:19:56 dist-test-slave-vnbh krb5kdc[18935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453996, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.195@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:19:56.453246 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.440930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:34225 (local address 127.31.119.254:38283)
0926 20:19:56.441090 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:56.441095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:56.442061 (+ 966us) server_negotiation.cc:407] Connection header received
0926 20:19:56.442927 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:56.442930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:56.442975 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:56.443052 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:56.444272 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.444815 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.446075 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.446286 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.446372 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.449410 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:56.449414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:56.449417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:56.449444 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:56.451006 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:56.451586 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:56.451590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:56.451591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:56.451710 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:56.452138 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:56.452141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:56.452143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:56.452352 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:56.452448 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.452988 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.453114 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:19:56.454138 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.441389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:38283 (local address 127.31.119.195:34225)
0926 20:19:56.441903 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:56.441947 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:56.442723 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:56.443219 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:56.443225 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:56.443579 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:56.444138 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.444151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.444947 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.444950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.445963 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.445973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.446418 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.446421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.446569 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.447268 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:19:56.447285 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:56.449231 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:56.451118 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:56.451123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:56.451136 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:56.451436 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:56.451870 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:19:56.451873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:19:56.451874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:19:56.452010 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:19:56.452450 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:56.452456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:56.452713 (+ 257us) client_negotiation.cc:769] Sending connection context
0926 20:19:56.452879 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 20:19:56.453155 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":103,"threads_started":1}
I20260926 20:19:56.455471 19439 heartbeater.cc:347] Connected to a master server at 127.31.119.254:38283
I20260926 20:19:56.455818 19439 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:56.456378 19439 heartbeater.cc:511] Master 127.31.119.254:38283 requested a full tablet report, sending...
I20260926 20:19:56.457573 18980 ts_manager.cc:194] Registered new tserver with Master: 64959d59003f4ca2b7bb4d0dae340364 (127.31.119.195:43823)
I20260926 20:19:56.458292 18980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.119.195@KRBTEST.COM'} at 127.31.119.195:34225
I20260926 20:19:56.462184 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:19:56.472491 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.464329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.31.119.254:38283)
0926 20:19:56.464511 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:56.464515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:56.464617 (+ 102us) server_negotiation.cc:407] Connection header received
0926 20:19:56.464704 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:56.464706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:56.464751 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:56.464884 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:56.465748 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.466321 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.467297 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.467517 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.467606 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.469155 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:56.469159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:56.469161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:56.469186 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:56.470622 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:56.471115 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:56.471118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:56.471120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:56.471209 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:56.471514 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:56.471517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:56.471518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:56.471787 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:56.471934 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.472178 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.472283 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 20:19:56.475831 18980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:19:56.479118 18980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:19:56.495512 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.489020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:43823 (local address 127.0.0.1:49336)
0926 20:19:56.489658 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:56.489675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:56.489772 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:56.490816 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:56.490820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:56.490838 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:56.490952 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.490962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.492376 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.492380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.493781 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.493790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.494864 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.494867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.494955 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.494990 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:19:56.495141 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 20:19:56.495329 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":125,"thread_start_us":86,"threads_started":1}
I20260926 20:19:56.495632 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.489538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.31.119.194:37851)
0926 20:19:56.490237 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:56.490252 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:56.490392 (+ 140us) server_negotiation.cc:407] Connection header received
0926 20:19:56.490473 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:56.490476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:56.490692 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:56.490854 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:56.491258 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.492049 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.494201 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.495014 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.495137 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.495278 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.495392 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.495486 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":101,"threads_started":1}
I20260926 20:19:56.495537 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.488746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:37851 (local address 127.0.0.1:41460)
0926 20:19:56.489567 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:56.489579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:56.489649 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:56.490985 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:56.490991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:56.491014 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:56.491124 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.491129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.492203 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.492210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.494077 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.494084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.495151 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.495155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.495253 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.495276 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:19:56.495361 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:19:56.495423 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":127,"thread_start_us":99,"threads_started":1}
I20260926 20:19:56.496498 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.489936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.31.119.195:43823)
0926 20:19:56.490358 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:56.490369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:56.490392 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:19:56.490466 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:56.490471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:56.490660 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:56.490823 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:56.491067 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.492192 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.493910 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.494732 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.494863 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.495013 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.496021 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.496349 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":109,"threads_started":1}
I20260926 20:19:56.496686 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.488675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:39309 (local address 127.0.0.1:60434)
0926 20:19:56.489344 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:56.489395 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:56.489649 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:56.490622 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:56.490627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:56.490650 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:56.490818 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.490827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.492333 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.492337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.494068 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.494080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.496288 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.496291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.496375 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.496399 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:19:56.496473 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:19:56.496546 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":162,"thread_start_us":95,"threads_started":1}
I20260926 20:19:56.497136 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.488876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.31.119.193:39309)
0926 20:19:56.490060 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:56.490078 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:56.490105 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:19:56.490190 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:56.490196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:56.490447 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:56.490622 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:56.490958 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.492192 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.495095 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.495948 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.496100 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.496228 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.496893 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.496995 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1057,"thread_start_us":67,"threads_started":1}
I20260926 20:19:56.499073 19239 tablet_service.cc:1467] Processing CreateTablet for tablet a8be6232dd104d83b5056804420ca3db (DEFAULT_TABLE table=test-table [id=63fe758bb385439d908ce853d5d2709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:56.500118 19239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8be6232dd104d83b5056804420ca3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:56.499753 19105 tablet_service.cc:1467] Processing CreateTablet for tablet a8be6232dd104d83b5056804420ca3db (DEFAULT_TABLE table=test-table [id=63fe758bb385439d908ce853d5d2709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:56.499753 19373 tablet_service.cc:1467] Processing CreateTablet for tablet a8be6232dd104d83b5056804420ca3db (DEFAULT_TABLE table=test-table [id=63fe758bb385439d908ce853d5d2709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:56.500804 19373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8be6232dd104d83b5056804420ca3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:56.500804 19105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8be6232dd104d83b5056804420ca3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:56.506633 19456 tablet_bootstrap.cc:492] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: Bootstrap starting.
I20260926 20:19:56.508404 19457 tablet_bootstrap.cc:492] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288: Bootstrap starting.
I20260926 20:19:56.509114 19456 tablet_bootstrap.cc:654] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:56.509492 19458 tablet_bootstrap.cc:492] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: Bootstrap starting.
I20260926 20:19:56.510752 19457 tablet_bootstrap.cc:654] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:56.510737 19456 log.cc:824] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:56.511587 19457 log.cc:824] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:56.511811 19458 tablet_bootstrap.cc:654] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:56.512755 19456 tablet_bootstrap.cc:492] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: No bootstrap required, opened a new log
I20260926 20:19:56.512969 19456 ts_tablet_manager.cc:1397] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:19:56.513006 19458 log.cc:824] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:56.514904 19458 tablet_bootstrap.cc:492] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: No bootstrap required, opened a new log
I20260926 20:19:56.514983 19457 tablet_bootstrap.cc:492] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288: No bootstrap required, opened a new log
I20260926 20:19:56.515130 19458 ts_tablet_manager.cc:1397] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:19:56.515164 19457 ts_tablet_manager.cc:1397] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260926 20:19:56.517097 19456 raft_consensus.cc:348] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853077bf34734e4780be6b4b86062288" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39309 } } peers { permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 } } peers { permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 } }
I20260926 20:19:56.517472 19456 raft_consensus.cc:374] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:56.517534 19456 raft_consensus.cc:729] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c9ddca231d48f1b332f56e87081077, State: Initialized, Role: FOLLOWER
I20260926 20:19:56.518090 19456 consensus_queue.cc:260] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853077bf34734e4780be6b4b86062288" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39309 } } peers { permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 } } peers { permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 } }
I20260926 20:19:56.518972 19458 raft_consensus.cc:348] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853077bf34734e4780be6b4b86062288" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39309 } } peers { permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 } } peers { permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 } }
I20260926 20:19:56.519227 19458 raft_consensus.cc:374] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:56.519263 19456 ts_tablet_manager.cc:1428] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:19:56.519289 19458 raft_consensus.cc:729] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64959d59003f4ca2b7bb4d0dae340364, State: Initialized, Role: FOLLOWER
I20260926 20:19:56.519429 19457 raft_consensus.cc:348] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853077bf34734e4780be6b4b86062288" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39309 } } peers { permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 } } peers { permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 } }
I20260926 20:19:56.519681 19457 raft_consensus.cc:374] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:56.519673 19305 heartbeater.cc:503] Master 127.31.119.254:38283 was elected leader, sending a full tablet report...
I20260926 20:19:56.519735 19457 raft_consensus.cc:729] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853077bf34734e4780be6b4b86062288, State: Initialized, Role: FOLLOWER
I20260926 20:19:56.519806 19458 consensus_queue.cc:260] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853077bf34734e4780be6b4b86062288" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39309 } } peers { permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 } } peers { permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 } }
I20260926 20:19:56.520155 19457 consensus_queue.cc:260] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853077bf34734e4780be6b4b86062288" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39309 } } peers { permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 } } peers { permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 } }
I20260926 20:19:56.520541 19439 heartbeater.cc:503] Master 127.31.119.254:38283 was elected leader, sending a full tablet report...
I20260926 20:19:56.520861 19458 ts_tablet_manager.cc:1428] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:19:56.521052 19457 ts_tablet_manager.cc:1428] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260926 20:19:56.521456 19171 heartbeater.cc:503] Master 127.31.119.254:38283 was elected leader, sending a full tablet report...
I20260926 20:19:56.548645 19464 raft_consensus.cc:482] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:19:56.548877 19464 raft_consensus.cc:504] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853077bf34734e4780be6b4b86062288" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39309 } } peers { permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 } } peers { permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 } }
I20260926 20:19:56.550628 19464 leader_election.cc:288] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5c9ddca231d48f1b332f56e87081077 (127.31.119.194:37851), 64959d59003f4ca2b7bb4d0dae340364 (127.31.119.195:43823)
I20260926 20:19:56.556176 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.551065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:40649 (local address 127.31.119.195:43823)
0926 20:19:56.551198 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:56.551202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:56.551217 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:19:56.551421 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:56.551424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:56.551483 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:56.551525 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:56.551982 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.552718 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.554741 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.555394 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.555534 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.555589 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.555981 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.556057 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:19:56.556176 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.550791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:42013 (local address 127.31.119.194:37851)
0926 20:19:56.550922 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:56.550928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:56.551133 (+ 205us) server_negotiation.cc:407] Connection header received
0926 20:19:56.551399 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:56.551403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:56.551462 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:56.551558 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:56.551873 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.552691 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.554741 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.555394 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.555534 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.555589 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.555954 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.556041 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:19:56.556243 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.550596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:37851 (local address 127.31.119.193:42013)
0926 20:19:56.551034 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:56.551049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:56.551245 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:56.551569 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:56.551572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:56.551602 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:56.551743 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.551749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.552864 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.552869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.554574 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.554580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.555572 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.555576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.555760 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.555794 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:19:56.555894 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:19:56.555995 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":180,"thread_start_us":154,"threads_started":1}
I20260926 20:19:56.556699 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.550594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:43823 (local address 127.31.119.193:40649)
0926 20:19:56.551106 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:56.551121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:56.551263 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:56.551667 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:56.551669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:56.551685 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:56.551852 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.551858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.552885 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.552890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.554567 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:56.554580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.555572 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.555576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:56.555840 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.555875 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:19:56.555958 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:19:56.556008 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":355,"spinlock_wait_cycles":78080,"thread_start_us":78,"threads_started":1}
I20260926 20:19:56.557085 19259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8be6232dd104d83b5056804420ca3db" candidate_uuid: "853077bf34734e4780be6b4b86062288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c9ddca231d48f1b332f56e87081077" is_pre_election: true
I20260926 20:19:56.557510 19259 raft_consensus.cc:2454] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853077bf34734e4780be6b4b86062288 in term 0.
I20260926 20:19:56.557487 19393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8be6232dd104d83b5056804420ca3db" candidate_uuid: "853077bf34734e4780be6b4b86062288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64959d59003f4ca2b7bb4d0dae340364" is_pre_election: true
I20260926 20:19:56.557755 19393 raft_consensus.cc:2454] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853077bf34734e4780be6b4b86062288 in term 0.
I20260926 20:19:56.558063 19058 leader_election.cc:302] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853077bf34734e4780be6b4b86062288, b5c9ddca231d48f1b332f56e87081077; no voters:
I20260926 20:19:56.558422 19464 raft_consensus.cc:2781] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:19:56.558502 19464 raft_consensus.cc:482] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:19:56.558534 19464 raft_consensus.cc:3037] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:56.559701 19464 raft_consensus.cc:504] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853077bf34734e4780be6b4b86062288" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39309 } } peers { permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 } } peers { permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 } }
I20260926 20:19:56.560114 19464 leader_election.cc:288] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [CANDIDATE]: Term 1 election: Requested vote from peers b5c9ddca231d48f1b332f56e87081077 (127.31.119.194:37851), 64959d59003f4ca2b7bb4d0dae340364 (127.31.119.195:43823)
I20260926 20:19:56.560515 19393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8be6232dd104d83b5056804420ca3db" candidate_uuid: "853077bf34734e4780be6b4b86062288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64959d59003f4ca2b7bb4d0dae340364"
I20260926 20:19:56.560623 19259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8be6232dd104d83b5056804420ca3db" candidate_uuid: "853077bf34734e4780be6b4b86062288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c9ddca231d48f1b332f56e87081077"
I20260926 20:19:56.560639 19393 raft_consensus.cc:3037] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:56.560742 19259 raft_consensus.cc:3037] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:56.562055 19259 raft_consensus.cc:2454] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853077bf34734e4780be6b4b86062288 in term 1.
I20260926 20:19:56.562055 19393 raft_consensus.cc:2454] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853077bf34734e4780be6b4b86062288 in term 1.
I20260926 20:19:56.562423 19061 leader_election.cc:302] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64959d59003f4ca2b7bb4d0dae340364, 853077bf34734e4780be6b4b86062288; no voters:
I20260926 20:19:56.562654 19464 raft_consensus.cc:2781] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:56.562915 19464 raft_consensus.cc:686] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [term 1 LEADER]: Becoming Leader. State: Replica: 853077bf34734e4780be6b4b86062288, State: Running, Role: LEADER
I20260926 20:19:56.563333 19464 consensus_queue.cc:237] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853077bf34734e4780be6b4b86062288" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39309 } } peers { permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 } } peers { permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 } }
I20260926 20:19:56.567341 18979 catalog_manager.cc:5495] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 853077bf34734e4780be6b4b86062288 (127.31.119.193). New cstate: current_term: 1 leader_uuid: "853077bf34734e4780be6b4b86062288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853077bf34734e4780be6b4b86062288" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 } health_report { overall_health: UNKNOWN } } }
I20260926 20:19:56.595902 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.592122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.31.119.193:39309)
0926 20:19:56.592266 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:56.592269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:56.592283 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:19:56.592364 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:56.592366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:56.592420 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:56.592512 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:19:56.592800 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.593381 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.594415 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.594624 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.594699 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.595036 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:19:56.595462 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:19:56.595554 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.595642 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.595696 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:19:56.602118 19393 raft_consensus.cc:1261] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364 [term 1 FOLLOWER]: Refusing update from remote peer 853077bf34734e4780be6b4b86062288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:56.602113 19259 raft_consensus.cc:1261] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077 [term 1 FOLLOWER]: Refusing update from remote peer 853077bf34734e4780be6b4b86062288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:56.602886 19464 consensus_queue.cc:1035] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64959d59003f4ca2b7bb4d0dae340364" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 43823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:56.603188 19467 consensus_queue.cc:1035] T a8be6232dd104d83b5056804420ca3db P 853077bf34734e4780be6b4b86062288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c9ddca231d48f1b332f56e87081077" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:56.612198 19472 mvcc.cc:205] Tried to move back new op lower bound from 7333699570074193920 to 7333699569927110656. Current Snapshot: MvccSnapshot[applied={T|T < 7333699570074193920}]
I20260926 20:19:56.612402 19471 mvcc.cc:205] Tried to move back new op lower bound from 7333699570074193920 to 7333699569927110656. Current Snapshot: MvccSnapshot[applied={T|T < 7333699570074193920}]
I20260926 20:19:56.620081 18980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:56.620301 18980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:56.623063 18980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8907cd9576f34e888262d7d0012ee02e: Sending DeleteTablet for 3 replicas of tablet a8be6232dd104d83b5056804420ca3db
I20260926 20:19:56.624001 19105 tablet_service.cc:1514] Processing DeleteTablet for tablet a8be6232dd104d83b5056804420ca3db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:56 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:60434
I20260926 20:19:56.624184 19373 tablet_service.cc:1514] Processing DeleteTablet for tablet a8be6232dd104d83b5056804420ca3db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:56 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:49336
I20260926 20:19:56.624686 19239 tablet_service.cc:1514] Processing DeleteTablet for tablet a8be6232dd104d83b5056804420ca3db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:56 UTC) from {username='kudu', principal='kudu/127.31.119.254@KRBTEST.COM'} at 127.0.0.1:41460
I20260926 20:19:56.624905 19480 tablet_replica.cc:330] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: stopping tablet replica
I20260926 20:19:56.625241 19481 tablet_replica.cc:330] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: stopping tablet replica
I20260926 20:19:56.625320 19480 raft_consensus.cc:2229] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:56.625723 19480 raft_consensus.cc:2258] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:56.625738 19481 raft_consensus.cc:2229] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:56.626197 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 19042
I20260926 20:19:56.626231 19481 raft_consensus.cc:2258] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:56.628261 19480 ts_tablet_manager.cc:1905] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:56.628810 19481 ts_tablet_manager.cc:1905] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:56.631610 19481 ts_tablet_manager.cc:1918] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:56.631665 19480 ts_tablet_manager.cc:1918] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:56.631764 19481 log.cc:1196] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/a8be6232dd104d83b5056804420ca3db
I20260926 20:19:56.631778 19480 log.cc:1196] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/a8be6232dd104d83b5056804420ca3db
I20260926 20:19:56.632115 19481 ts_tablet_manager.cc:1939] T a8be6232dd104d83b5056804420ca3db P b5c9ddca231d48f1b332f56e87081077: Deleting consensus metadata
I20260926 20:19:56.632125 19480 ts_tablet_manager.cc:1939] T a8be6232dd104d83b5056804420ca3db P 64959d59003f4ca2b7bb4d0dae340364: Deleting consensus metadata
I20260926 20:19:56.633594 18964 catalog_manager.cc:4841] TS b5c9ddca231d48f1b332f56e87081077 (127.31.119.194:37851): tablet a8be6232dd104d83b5056804420ca3db (table test-table [id=63fe758bb385439d908ce853d5d2709d]) successfully deleted
I20260926 20:19:56.633592 18967 catalog_manager.cc:4841] TS 64959d59003f4ca2b7bb4d0dae340364 (127.31.119.195:43823): tablet a8be6232dd104d83b5056804420ca3db (table test-table [id=63fe758bb385439d908ce853d5d2709d]) successfully deleted
W20260926 20:19:56.635456 18965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:39309 (error 108)
I20260926 20:19:56.636412 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 19176
I20260926 20:19:56.636711 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.636289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:39309 (local address 127.0.0.1:60452)
0926 20:19:56.636451 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:56.636539 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:39309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 20:19:56.637490 18965 catalog_manager.cc:4584] TS 853077bf34734e4780be6b4b86062288 (127.31.119.193:39309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8be6232dd104d83b5056804420ca3db: Network error: Client connection negotiation failed: client connection to 127.31.119.193:39309: connect: Connection refused (error 111)
I20260926 20:19:56.642511 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 19310
I20260926 20:19:56.648337 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 18948
2026-09-26T20:19:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2360 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T20:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:19:56Z Disabled control of system clock
I20260926 20:19:56.680929 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:42377
--webserver_interface=127.31.119.254
--webserver_port=0
--builtin_ntp_servers=127.31.119.212:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:42377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:19:56.796340 19486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:56.796664 19486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:56.796754 19486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:56.801396 19486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:19:56.801482 19486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:56.801510 19486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:56.801538 19486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:19:56.801566 19486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:19:56.806768 19486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:42377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:42377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:56.807981 19486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:56.809527 19486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:56.818670 19494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:56.818696 19491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:56.818666 19492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:56.818943 19486 server_base.cc:1034] running on GCE node
I20260926 20:19:56.819725 19486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:56.820888 19486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:56.822057 19486 hybrid_clock.cc:648] HybridClock initialized: now 1790453996822024 us; error 44 us; skew 500 ppm
I20260926 20:19:56.824014 19486 webserver.cc:460] Webserver started at http://127.31.119.254:40433/ using document root <none> and password file <none>
I20260926 20:19:56.824620 19486 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:56.824703 19486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:56.824932 19486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:56.827153 19486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "e6bb495d65774677a61d265553a8189c"
format_stamp: "Formatted at 2026-09-26 20:19:56 on dist-test-slave-vnbh"
server_key: "0d97df2f7eb22aab25a60ef1e8188f8e"
server_key_iv: "aa6cd5a54b6672dfbe61cce79c55ffd3"
server_key_version: "encryptionkey@0"
I20260926 20:19:56.827733 19486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "e6bb495d65774677a61d265553a8189c"
format_stamp: "Formatted at 2026-09-26 20:19:56 on dist-test-slave-vnbh"
server_key: "0d97df2f7eb22aab25a60ef1e8188f8e"
server_key_iv: "aa6cd5a54b6672dfbe61cce79c55ffd3"
server_key_version: "encryptionkey@0"
I20260926 20:19:56.831488 19486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:19:56.834059 19500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:56.835624 19486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:56.835747 19486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "e6bb495d65774677a61d265553a8189c"
format_stamp: "Formatted at 2026-09-26 20:19:56 on dist-test-slave-vnbh"
server_key: "0d97df2f7eb22aab25a60ef1e8188f8e"
server_key_iv: "aa6cd5a54b6672dfbe61cce79c55ffd3"
server_key_version: "encryptionkey@0"
I20260926 20:19:56.835847 19486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:56.856225 19486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:56.856947 19486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:56.857195 19486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:56.864765 19486 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:42377
I20260926 20:19:56.864792 19552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:42377 every 8 connection(s)
I20260926 20:19:56.865854 19486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:19:56.867463 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 19486
I20260926 20:19:56.867619 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:19:56.867975 32223 external_mini_cluster.cc:1428] Setting key 27bdf505549800810f8c24dbc232a5a4
I20260926 20:19:56.869719 19553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:56.875701 19553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c: Bootstrap starting.
I20260926 20:19:56.878162 19553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:56.878165 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:56.869544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.31.119.254:42377)
0926 20:19:56.870324 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:56.870341 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:56.870381 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:19:56.871252 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:56.871289 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:56.871723 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:56.872105 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:56.872576 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.873653 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.874585 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:56.875084 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:56.875279 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:56.876249 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:56.876254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:56.876268 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:56.876674 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:56.876743 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:56.876763 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:56.876880 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:56.877015 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 20:19:56.877352 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":148,"threads_started":1}
I20260926 20:19:56.879196 19553 log.cc:824] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:56.881273 19553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c: No bootstrap required, opened a new log
I20260926 20:19:56.885049 19553 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bb495d65774677a61d265553a8189c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42377 } }
I20260926 20:19:56.885332 19553 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:56.885408 19553 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6bb495d65774677a61d265553a8189c, State: Initialized, Role: FOLLOWER
I20260926 20:19:56.885787 19553 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bb495d65774677a61d265553a8189c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42377 } }
I20260926 20:19:56.885911 19553 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:19:56.885957 19553 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:19:56.886027 19553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:56.886914 19553 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bb495d65774677a61d265553a8189c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42377 } }
I20260926 20:19:56.887210 19553 leader_election.cc:302] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6bb495d65774677a61d265553a8189c; no voters:
I20260926 20:19:56.887470 19553 leader_election.cc:288] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:19:56.887607 19558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:56.887943 19558 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [term 1 LEADER]: Becoming Leader. State: Replica: e6bb495d65774677a61d265553a8189c, State: Running, Role: LEADER
I20260926 20:19:56.888252 19558 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bb495d65774677a61d265553a8189c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42377 } }
I20260926 20:19:56.888839 19553 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:19:56.889943 19560 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6bb495d65774677a61d265553a8189c. Latest consensus state: current_term: 1 leader_uuid: "e6bb495d65774677a61d265553a8189c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bb495d65774677a61d265553a8189c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42377 } } }
I20260926 20:19:56.890064 19560 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:56.890344 19559 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6bb495d65774677a61d265553a8189c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bb495d65774677a61d265553a8189c" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 42377 } } }
I20260926 20:19:56.890452 19559 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:56.890535 19567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:19:56.893945 19567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:19:56.899250 19567 catalog_manager.cc:1347] Generated new cluster ID: e43068ad744c4128a28c8a585950df90
I20260926 20:19:56.899322 19567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:19:56.920459 19567 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:19:56.921572 19567 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:19:56.929230 19567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c: Generated new TSK 0
I20260926 20:19:56.930047 19567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:19:56.937103 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.193:0
--local_ip_for_outbound_sockets=127.31.119.193
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42377
--builtin_ntp_servers=127.31.119.212:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:19:57.053505 19577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:57.053797 19577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:57.053836 19577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:57.058013 19577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:57.058079 19577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:57.058136 19577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.193
I20260926 20:19:57.063000 19577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.119.193
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.193
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:57.064155 19577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:57.065666 19577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:19:57.076443 19577 server_base.cc:1034] running on GCE node
W20260926 20:19:57.076363 19582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:57.076370 19583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:57.076370 19585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:57.077037 19577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:57.077855 19577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:57.079041 19577 hybrid_clock.cc:648] HybridClock initialized: now 1790453997079004 us; error 51 us; skew 500 ppm
I20260926 20:19:57.081130 19577 webserver.cc:460] Webserver started at http://127.31.119.193:44935/ using document root <none> and password file <none>
I20260926 20:19:57.081796 19577 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:57.081863 19577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:57.082089 19577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:57.084390 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/data/instance:
uuid: "39b74f6fc9784f0e8adf93759c13584a"
format_stamp: "Formatted at 2026-09-26 20:19:57 on dist-test-slave-vnbh"
server_key: "d1bcc87f8500bf7b939c7981d429b3eb"
server_key_iv: "344d8193e68aa5b70769604593611fcd"
server_key_version: "encryptionkey@0"
I20260926 20:19:57.085059 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance:
uuid: "39b74f6fc9784f0e8adf93759c13584a"
format_stamp: "Formatted at 2026-09-26 20:19:57 on dist-test-slave-vnbh"
server_key: "d1bcc87f8500bf7b939c7981d429b3eb"
server_key_iv: "344d8193e68aa5b70769604593611fcd"
server_key_version: "encryptionkey@0"
I20260926 20:19:57.089164 19577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:19:57.092263 19591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:57.093849 19577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:57.094018 19577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/wal
uuid: "39b74f6fc9784f0e8adf93759c13584a"
format_stamp: "Formatted at 2026-09-26 20:19:57 on dist-test-slave-vnbh"
server_key: "d1bcc87f8500bf7b939c7981d429b3eb"
server_key_iv: "344d8193e68aa5b70769604593611fcd"
server_key_version: "encryptionkey@0"
I20260926 20:19:57.094151 19577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:57.120759 19577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:57.121627 19577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:57.121912 19577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:57.122545 19577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:57.123536 19577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:57.123611 19577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:57.123680 19577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:57.123714 19577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:57.133445 19577 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.193:35373
I20260926 20:19:57.133464 19704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.193:35373 every 8 connection(s)
I20260926 20:19:57.134403 19577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/data/info.pb
I20260926 20:19:57.134620 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 19577
I20260926 20:19:57.134776 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-0/wal/instance
I20260926 20:19:57.135156 32223 external_mini_cluster.cc:1428] Setting key fb96e255af2a9551b9b653abfe0399c1
I20260926 20:19:57.137607 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.194:0
--local_ip_for_outbound_sockets=127.31.119.194
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42377
--builtin_ntp_servers=127.31.119.212:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:19:57.146456 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.136599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.193:46497 (local address 127.31.119.254:42377)
0926 20:19:57.136741 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.136746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.138074 (+ 1328us) server_negotiation.cc:407] Connection header received
0926 20:19:57.139819 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.139823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.139879 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.139989 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:57.141269 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.141954 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.143964 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.144195 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.144306 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.145866 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:57.145869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:57.145871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:57.145885 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:57.145907 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:57.145915 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:57.145977 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.146241 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.146328 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:19:57.147933 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.137131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:42377 (local address 127.31.119.193:46497)
0926 20:19:57.137904 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.137956 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.139554 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.140195 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:57.140207 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:57.140332 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:57.141080 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.141104 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.142114 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.142122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.143816 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.143831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.144387 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.144395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.144676 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.145493 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:19:57.145497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:57.145565 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:19:57.145666 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:57.146012 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:57.146025 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:57.146034 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:19:57.146254 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 20:19:57.146633 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":157,"threads_started":1}
I20260926 20:19:57.149485 19705 heartbeater.cc:347] Connected to a master server at 127.31.119.254:42377
I20260926 20:19:57.149755 19705 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:57.150377 19705 heartbeater.cc:511] Master 127.31.119.254:42377 requested a full tablet report, sending...
I20260926 20:19:57.152053 19517 ts_manager.cc:194] Registered new tserver with Master: 39b74f6fc9784f0e8adf93759c13584a (127.31.119.193:35373)
I20260926 20:19:57.153223 19517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.193:46497
W20260926 20:19:57.259562 19708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:57.259922 19708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:57.259999 19708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:57.264274 19708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:57.264367 19708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:57.264464 19708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.194
I20260926 20:19:57.269421 19708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.119.194
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.194
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:57.270684 19708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:57.272179 19708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:19:57.282578 19708 server_base.cc:1034] running on GCE node
W20260926 20:19:57.282572 19713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:57.282572 19714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:57.282902 19716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:57.283311 19708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:57.283995 19708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:57.285166 19708 hybrid_clock.cc:648] HybridClock initialized: now 1790453997285152 us; error 55 us; skew 500 ppm
I20260926 20:19:57.287091 19708 webserver.cc:460] Webserver started at http://127.31.119.194:39657/ using document root <none> and password file <none>
I20260926 20:19:57.287634 19708 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:57.287695 19708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:57.287921 19708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:57.290220 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/data/instance:
uuid: "b85d9e96a28c4336aac0d0c495bf9822"
format_stamp: "Formatted at 2026-09-26 20:19:57 on dist-test-slave-vnbh"
server_key: "48b57f416cdf6c322832007e87d0df14"
server_key_iv: "f377efc944ce9426e509fe6e462dc8cd"
server_key_version: "encryptionkey@0"
I20260926 20:19:57.290807 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance:
uuid: "b85d9e96a28c4336aac0d0c495bf9822"
format_stamp: "Formatted at 2026-09-26 20:19:57 on dist-test-slave-vnbh"
server_key: "48b57f416cdf6c322832007e87d0df14"
server_key_iv: "f377efc944ce9426e509fe6e462dc8cd"
server_key_version: "encryptionkey@0"
I20260926 20:19:57.294639 19708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:19:57.297531 19722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:57.299062 19708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:19:57.299165 19708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/wal
uuid: "b85d9e96a28c4336aac0d0c495bf9822"
format_stamp: "Formatted at 2026-09-26 20:19:57 on dist-test-slave-vnbh"
server_key: "48b57f416cdf6c322832007e87d0df14"
server_key_iv: "f377efc944ce9426e509fe6e462dc8cd"
server_key_version: "encryptionkey@0"
I20260926 20:19:57.299264 19708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:57.320317 19708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:57.321151 19708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:57.321393 19708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:57.321998 19708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:57.322956 19708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:57.323001 19708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:57.323041 19708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:57.323056 19708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:57.332590 19708 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.194:37085
I20260926 20:19:57.332614 19835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.194:37085 every 8 connection(s)
I20260926 20:19:57.333645 19708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/data/info.pb
I20260926 20:19:57.335577 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 19708
I20260926 20:19:57.335724 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/wal/instance
I20260926 20:19:57.336082 32223 external_mini_cluster.cc:1428] Setting key 629f556b46f5461802182a54adfaf53e
I20260926 20:19:57.338351 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.119.195:0
--local_ip_for_outbound_sockets=127.31.119.195
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42377
--builtin_ntp_servers=127.31.119.212:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:19:57.344966 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.335913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:40651 (local address 127.31.119.254:42377)
0926 20:19:57.336088 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.336093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.337419 (+ 1326us) server_negotiation.cc:407] Connection header received
0926 20:19:57.339009 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.339015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.339079 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.339178 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:57.340489 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.341125 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.342674 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.342914 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.343046 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.344332 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:57.344336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:57.344339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:57.344361 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:57.344385 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:57.344393 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:57.344478 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.344740 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.344811 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:19:57.345923 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.336434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:42377 (local address 127.31.119.194:40651)
0926 20:19:57.337218 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.337278 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.338782 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.339349 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:57.339357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:57.339447 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:57.340324 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.340348 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.341264 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.341267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.342366 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.342376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.343093 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.343097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.343289 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.344011 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:19:57.344012 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:57.344056 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:19:57.344128 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:57.344480 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:57.344493 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:57.344499 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:19:57.344661 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 20:19:57.344974 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":153,"threads_started":1}
I20260926 20:19:57.347317 19836 heartbeater.cc:347] Connected to a master server at 127.31.119.254:42377
I20260926 20:19:57.347606 19836 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:57.348172 19836 heartbeater.cc:511] Master 127.31.119.254:42377 requested a full tablet report, sending...
I20260926 20:19:57.349313 19517 ts_manager.cc:194] Registered new tserver with Master: b85d9e96a28c4336aac0d0c495bf9822 (127.31.119.194:37085)
I20260926 20:19:57.350085 19517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.194:40651
W20260926 20:19:57.462299 19839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:57.462601 19839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:57.462642 19839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:57.467098 19839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:57.467165 19839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:57.467224 19839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.119.195
I20260926 20:19:57.472404 19839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.119.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.119.195
--webserver_port=0
--tserver_master_addrs=127.31.119.254:42377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.119.195
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:57.473757 19839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:57.475263 19839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:57.487301 19845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:57.487361 19844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:57.487653 19847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:57.487658 19839 server_base.cc:1034] running on GCE node
I20260926 20:19:57.488272 19839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:57.489174 19839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:57.490376 19839 hybrid_clock.cc:648] HybridClock initialized: now 1790453997490337 us; error 64 us; skew 500 ppm
I20260926 20:19:57.493083 19839 webserver.cc:460] Webserver started at http://127.31.119.195:46505/ using document root <none> and password file <none>
I20260926 20:19:57.493945 19839 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:57.494043 19839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:57.494299 19839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:57.497437 19839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/data/instance:
uuid: "7a5cac23fd094d5a87c823773e34973c"
format_stamp: "Formatted at 2026-09-26 20:19:57 on dist-test-slave-vnbh"
server_key: "643a43f1eb5f2f40e66116bf57debe10"
server_key_iv: "d176251a8d31c69d753026f8c9bb6f24"
server_key_version: "encryptionkey@0"
I20260926 20:19:57.498240 19839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance:
uuid: "7a5cac23fd094d5a87c823773e34973c"
format_stamp: "Formatted at 2026-09-26 20:19:57 on dist-test-slave-vnbh"
server_key: "643a43f1eb5f2f40e66116bf57debe10"
server_key_iv: "d176251a8d31c69d753026f8c9bb6f24"
server_key_version: "encryptionkey@0"
I20260926 20:19:57.503984 19839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:19:57.507553 19853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:57.509152 19839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:19:57.509263 19839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/wal
uuid: "7a5cac23fd094d5a87c823773e34973c"
format_stamp: "Formatted at 2026-09-26 20:19:57 on dist-test-slave-vnbh"
server_key: "643a43f1eb5f2f40e66116bf57debe10"
server_key_iv: "d176251a8d31c69d753026f8c9bb6f24"
server_key_version: "encryptionkey@0"
I20260926 20:19:57.509459 19839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:57.529215 19839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:57.530148 19839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:57.530372 19839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:57.531044 19839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:19:57.532061 19839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:19:57.532109 19839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:57.532148 19839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:19:57.532204 19839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:57.542279 19839 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.195:44717
I20260926 20:19:57.542325 19966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.195:44717 every 8 connection(s)
I20260926 20:19:57.543287 19839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/data/info.pb
I20260926 20:19:57.546247 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 19839
I20260926 20:19:57.546382 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/wal/instance
I20260926 20:19:57.546694 32223 external_mini_cluster.cc:1428] Setting key 4e1069dbc175056acc4b3c957df4943a
I20260926 20:19:57.553930 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.545501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:60193 (local address 127.31.119.254:42377)
0926 20:19:57.545671 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.545675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.547115 (+ 1440us) server_negotiation.cc:407] Connection header received
0926 20:19:57.548876 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.548879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.548931 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.549001 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:57.550030 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.550613 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.551825 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.552060 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.552177 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.553376 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:57.553383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:57.553387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:57.553402 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:57.553430 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:57.553438 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:57.553516 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.553694 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.553762 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 20:19:57.554898 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.546003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:42377 (local address 127.31.119.195:60193)
0926 20:19:57.546912 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.546977 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.548645 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.549195 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:57.549203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:57.549353 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:19:57.549886 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.549899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.550759 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.550764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.551692 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.551701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.552222 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.552226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.552409 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.553072 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:19:57.553074 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:19:57.553116 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:19:57.553186 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:19:57.553529 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:19:57.553542 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:19:57.553553 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 20:19:57.553680 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 20:19:57.553983 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
I20260926 20:19:57.556183 19967 heartbeater.cc:347] Connected to a master server at 127.31.119.254:42377
I20260926 20:19:57.556433 19967 heartbeater.cc:464] Registering TS with master...
I20260926 20:19:57.556962 19967 heartbeater.cc:511] Master 127.31.119.254:42377 requested a full tablet report, sending...
I20260926 20:19:57.558215 19517 ts_manager.cc:194] Registered new tserver with Master: 7a5cac23fd094d5a87c823773e34973c (127.31.119.195:44717)
I20260926 20:19:57.558766 19517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.119.195:60193
I20260926 20:19:57.561040 32223 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:19:57.567687 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.563155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.31.119.254:42377)
0926 20:19:57.563353 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.563357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.563478 (+ 121us) server_negotiation.cc:407] Connection header received
0926 20:19:57.563550 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.563552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.563598 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.563671 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:57.564052 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.564590 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.565675 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.565913 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.566008 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.567115 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:57.567118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:57.567120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:19:57.567134 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:57.567156 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:19:57.567164 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:57.567240 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.567392 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.567497 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 20:19:57.571404 19517 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:19:57.574903 19517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:19:57.591763 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.586149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35373 (local address 127.0.0.1:44638)
0926 20:19:57.586447 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.586461 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.586545 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.587266 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:57.587269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:57.587296 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:57.587390 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.587399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.588477 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.588481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.589904 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.589914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.591071 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.591075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.591199 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.591246 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:19:57.591387 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 20:19:57.591542 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":57,"thread_start_us":94,"threads_started":1}
I20260926 20:19:57.591836 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.584820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:44717 (local address 127.0.0.1:57586)
0926 20:19:57.585252 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.585348 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.585512 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.586757 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:57.586764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:57.586791 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:57.586964 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.586972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.588746 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.588749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.590244 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.590257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.591412 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.591415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.591505 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.591528 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:19:57.591597 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:19:57.591673 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":60,"thread_start_us":116,"threads_started":1}
I20260926 20:19:57.592358 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.585211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.31.119.195:44717)
0926 20:19:57.586167 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.586180 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.586212 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:19:57.586317 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.586324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.586564 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.587140 (+ 576us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:57.587321 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.588614 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.590394 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.591225 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.591402 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.591743 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.591813 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.592233 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":224,"threads_started":1}
I20260926 20:19:57.592523 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.584882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:37085 (local address 127.0.0.1:34386)
0926 20:19:57.585872 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.585886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.585973 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.586964 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:57.586967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:57.586983 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:57.587093 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.587099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.588724 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.588731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.590643 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.590652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.592067 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.592074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.592190 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.592229 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 20:19:57.592307 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:19:57.592377 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":49,"thread_start_us":99,"threads_started":1}
I20260926 20:19:57.593034 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.585857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.31.119.194:37085)
0926 20:19:57.586358 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.586370 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.586391 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:19:57.586461 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.586465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.586623 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.586799 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:57.587223 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.588569 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.590797 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.591633 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.591778 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.591986 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.592729 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.592865 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":154,"threads_started":1}
I20260926 20:19:57.593516 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.586516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.31.119.193:35373)
0926 20:19:57.586855 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.586868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.586893 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:19:57.586960 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.586964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.587115 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.587275 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:57.587486 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.588301 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.590045 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.590930 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.591064 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.591266 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.593154 (+ 1888us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.593338 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":146,"threads_started":1}
I20260926 20:19:57.595431 19901 tablet_service.cc:1467] Processing CreateTablet for tablet 5fda77eba7244a548c40ce7bd21c63f0 (DEFAULT_TABLE table=test-table [id=ad3411c9456548f7aa863e5efaa3d0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:57.596480 19901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fda77eba7244a548c40ce7bd21c63f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:57.596068 19770 tablet_service.cc:1467] Processing CreateTablet for tablet 5fda77eba7244a548c40ce7bd21c63f0 (DEFAULT_TABLE table=test-table [id=ad3411c9456548f7aa863e5efaa3d0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:57.596247 19639 tablet_service.cc:1467] Processing CreateTablet for tablet 5fda77eba7244a548c40ce7bd21c63f0 (DEFAULT_TABLE table=test-table [id=ad3411c9456548f7aa863e5efaa3d0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:19:57.597231 19639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fda77eba7244a548c40ce7bd21c63f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:57.597608 19770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fda77eba7244a548c40ce7bd21c63f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:57.603591 19984 tablet_bootstrap.cc:492] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: Bootstrap starting.
I20260926 20:19:57.605634 19985 tablet_bootstrap.cc:492] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a: Bootstrap starting.
I20260926 20:19:57.606401 19984 tablet_bootstrap.cc:654] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:57.607366 19984 log.cc:824] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:57.608176 19985 tablet_bootstrap.cc:654] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:57.609118 19985 log.cc:824] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:57.609726 19986 tablet_bootstrap.cc:492] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: Bootstrap starting.
I20260926 20:19:57.610124 19984 tablet_bootstrap.cc:492] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: No bootstrap required, opened a new log
I20260926 20:19:57.610352 19984 ts_tablet_manager.cc:1397] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260926 20:19:57.611755 19985 tablet_bootstrap.cc:492] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a: No bootstrap required, opened a new log
I20260926 20:19:57.611805 19986 tablet_bootstrap.cc:654] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:57.611970 19985 ts_tablet_manager.cc:1397] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 20:19:57.612644 19986 log.cc:824] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:57.615211 19986 tablet_bootstrap.cc:492] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: No bootstrap required, opened a new log
I20260926 20:19:57.615041 19984 raft_consensus.cc:348] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } }
I20260926 20:19:57.615410 19984 raft_consensus.cc:374] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:57.615427 19986 ts_tablet_manager.cc:1397] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:19:57.615464 19984 raft_consensus.cc:729] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5cac23fd094d5a87c823773e34973c, State: Initialized, Role: FOLLOWER
I20260926 20:19:57.616055 19984 consensus_queue.cc:260] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } }
I20260926 20:19:57.616732 19985 raft_consensus.cc:348] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } }
I20260926 20:19:57.617018 19985 raft_consensus.cc:374] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:57.617075 19985 raft_consensus.cc:729] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b74f6fc9784f0e8adf93759c13584a, State: Initialized, Role: FOLLOWER
I20260926 20:19:57.617204 19984 ts_tablet_manager.cc:1428] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260926 20:19:57.617636 19967 heartbeater.cc:503] Master 127.31.119.254:42377 was elected leader, sending a full tablet report...
I20260926 20:19:57.617610 19985 consensus_queue.cc:260] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } }
I20260926 20:19:57.618347 19705 heartbeater.cc:503] Master 127.31.119.254:42377 was elected leader, sending a full tablet report...
I20260926 20:19:57.618705 19985 ts_tablet_manager.cc:1428] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260926 20:19:57.620074 19986 raft_consensus.cc:348] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } }
I20260926 20:19:57.620388 19986 raft_consensus.cc:374] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:57.620460 19986 raft_consensus.cc:729] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85d9e96a28c4336aac0d0c495bf9822, State: Initialized, Role: FOLLOWER
I20260926 20:19:57.620949 19986 consensus_queue.cc:260] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } }
I20260926 20:19:57.621807 19836 heartbeater.cc:503] Master 127.31.119.254:42377 was elected leader, sending a full tablet report...
I20260926 20:19:57.622047 19986 ts_tablet_manager.cc:1428] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
W20260926 20:19:57.635857 19706 tablet.cc:2369] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:19:57.794373 19968 tablet.cc:2369] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:19:57.835080 19837 tablet.cc:2369] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:19:57.842746 19990 raft_consensus.cc:482] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:19:57.842986 19990 raft_consensus.cc:504] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } }
I20260926 20:19:57.844350 19990 leader_election.cc:288] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b85d9e96a28c4336aac0d0c495bf9822 (127.31.119.194:37085), 39b74f6fc9784f0e8adf93759c13584a (127.31.119.193:35373)
I20260926 20:19:57.845629 19992 raft_consensus.cc:482] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:19:57.845960 19992 raft_consensus.cc:504] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } }
I20260926 20:19:57.847918 19992 leader_election.cc:288] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a5cac23fd094d5a87c823773e34973c (127.31.119.195:44717), 39b74f6fc9784f0e8adf93759c13584a (127.31.119.193:35373)
I20260926 20:19:57.849170 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.844502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.194:37085 (local address 127.31.119.195:38061)
0926 20:19:57.844678 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.844696 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.844798 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.845186 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:57.845189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:57.845220 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:57.845355 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.845362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.846500 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.846503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.847680 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.847690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.848734 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.848737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.848844 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.848867 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:19:57.848938 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:19:57.849012 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 20:19:57.849340 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.845058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:54897 (local address 127.31.119.193:35373)
0926 20:19:57.845216 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.845220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.845422 (+ 202us) server_negotiation.cc:407] Connection header received
0926 20:19:57.845548 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.845550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.845602 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.845688 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:57.845997 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.846691 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.847994 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.848645 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.848732 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.848776 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.849091 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.849165 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:19:57.849731 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.844893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35373 (local address 127.31.119.195:54897)
0926 20:19:57.845273 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.845329 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.845460 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.845716 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:57.845719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:57.845739 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:57.845876 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.845883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.846832 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.846835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.847846 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.847856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.848866 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.848869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.848966 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.848987 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:19:57.849532 (+ 545us) client_negotiation.cc:240] Negotiation successful
0926 20:19:57.849601 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":126,"threads_started":1}
I20260926 20:19:57.849807 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.844660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.195:38061 (local address 127.31.119.194:37085)
0926 20:19:57.844883 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.844887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.844904 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:19:57.844967 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.844970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.845021 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.845129 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:57.845523 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.846374 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.847839 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.848481 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.848589 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.848646 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.849600 (+ 954us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.849673 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":115}
I20260926 20:19:57.850322 19790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fda77eba7244a548c40ce7bd21c63f0" candidate_uuid: "7a5cac23fd094d5a87c823773e34973c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b85d9e96a28c4336aac0d0c495bf9822" is_pre_election: true
I20260926 20:19:57.850567 19790 raft_consensus.cc:2454] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a5cac23fd094d5a87c823773e34973c in term 0.
I20260926 20:19:57.851009 19659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fda77eba7244a548c40ce7bd21c63f0" candidate_uuid: "7a5cac23fd094d5a87c823773e34973c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b74f6fc9784f0e8adf93759c13584a" is_pre_election: true
I20260926 20:19:57.851053 19856 leader_election.cc:302] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a5cac23fd094d5a87c823773e34973c, b85d9e96a28c4336aac0d0c495bf9822; no voters:
I20260926 20:19:57.851320 19659 raft_consensus.cc:2454] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a5cac23fd094d5a87c823773e34973c in term 0.
I20260926 20:19:57.851342 19990 raft_consensus.cc:2781] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:19:57.851420 19990 raft_consensus.cc:482] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:19:57.851480 19990 raft_consensus.cc:3037] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:57.852695 19990 raft_consensus.cc:504] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } }
I20260926 20:19:57.853078 19990 leader_election.cc:288] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [CANDIDATE]: Term 1 election: Requested vote from peers b85d9e96a28c4336aac0d0c495bf9822 (127.31.119.194:37085), 39b74f6fc9784f0e8adf93759c13584a (127.31.119.193:35373)
I20260926 20:19:57.853199 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.847413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35373 (local address 127.31.119.194:48901)
0926 20:19:57.848138 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.848151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.848243 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.848731 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:57.848733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:57.848750 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:57.848859 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.848865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.850132 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.850135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.851810 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.851821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.852755 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.852759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.852859 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.852884 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:19:57.852967 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:19:57.853038 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":80,"threads_started":1}
I20260926 20:19:57.853248 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.847736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:48901 (local address 127.31.119.193:35373)
0926 20:19:57.848511 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.848517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.848532 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:19:57.848574 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.848577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.848617 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.848962 (+ 345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:57.849046 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.850005 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.851941 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.852592 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.852739 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.852785 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.852998 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.853088 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":82,"threads_started":1}
I20260926 20:19:57.853546 19790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fda77eba7244a548c40ce7bd21c63f0" candidate_uuid: "7a5cac23fd094d5a87c823773e34973c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b85d9e96a28c4336aac0d0c495bf9822"
I20260926 20:19:57.853490 19659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fda77eba7244a548c40ce7bd21c63f0" candidate_uuid: "7a5cac23fd094d5a87c823773e34973c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b74f6fc9784f0e8adf93759c13584a"
I20260926 20:19:57.853667 19659 raft_consensus.cc:3037] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:57.853662 19790 raft_consensus.cc:3037] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:57.853789 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.847662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.119.194:37007 (local address 127.31.119.195:44717)
0926 20:19:57.847794 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.847798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.847999 (+ 201us) server_negotiation.cc:407] Connection header received
0926 20:19:57.848166 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.848169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.848209 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.848252 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:19:57.848628 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.849327 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.852028 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.852603 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.852742 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.852785 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.853604 (+ 819us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.853677 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 20:19:57.854135 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.847383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.195:44717 (local address 127.31.119.194:37007)
0926 20:19:57.847933 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.847949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.848060 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.848361 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:19:57.848364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:19:57.848380 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:19:57.848508 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.848514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.849473 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.849478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.851951 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:19:57.851956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.853346 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.853351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:19:57.853459 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.853484 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:19:57.853907 (+ 423us) client_negotiation.cc:240] Negotiation successful
0926 20:19:57.853983 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260926 20:19:57.854533 19658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fda77eba7244a548c40ce7bd21c63f0" candidate_uuid: "b85d9e96a28c4336aac0d0c495bf9822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b74f6fc9784f0e8adf93759c13584a" is_pre_election: true
I20260926 20:19:57.854668 19921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fda77eba7244a548c40ce7bd21c63f0" candidate_uuid: "b85d9e96a28c4336aac0d0c495bf9822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5cac23fd094d5a87c823773e34973c" is_pre_election: true
I20260926 20:19:57.854934 19921 raft_consensus.cc:2379] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b85d9e96a28c4336aac0d0c495bf9822 in current term 1: Already voted for candidate 7a5cac23fd094d5a87c823773e34973c in this term.
I20260926 20:19:57.854986 19659 raft_consensus.cc:2454] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a5cac23fd094d5a87c823773e34973c in term 1.
I20260926 20:19:57.854983 19790 raft_consensus.cc:2454] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a5cac23fd094d5a87c823773e34973c in term 1.
I20260926 20:19:57.855350 19856 leader_election.cc:302] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b74f6fc9784f0e8adf93759c13584a, 7a5cac23fd094d5a87c823773e34973c; no voters:
I20260926 20:19:57.855516 19990 raft_consensus.cc:2781] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:57.855511 19723 leader_election.cc:302] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b85d9e96a28c4336aac0d0c495bf9822; no voters: 39b74f6fc9784f0e8adf93759c13584a, 7a5cac23fd094d5a87c823773e34973c
I20260926 20:19:57.855747 19990 raft_consensus.cc:686] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 1 LEADER]: Becoming Leader. State: Replica: 7a5cac23fd094d5a87c823773e34973c, State: Running, Role: LEADER
I20260926 20:19:57.855836 19992 raft_consensus.cc:2726] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 20:19:57.856164 19990 consensus_queue.cc:237] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } }
I20260926 20:19:57.859777 19516 catalog_manager.cc:5495] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a5cac23fd094d5a87c823773e34973c (127.31.119.195). New cstate: current_term: 1 leader_uuid: "7a5cac23fd094d5a87c823773e34973c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5cac23fd094d5a87c823773e34973c" member_type: VOTER last_known_addr { host: "127.31.119.195" port: 44717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 } health_report { overall_health: UNKNOWN } } }
I20260926 20:19:57.889652 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:57.885724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.31.119.195:44717)
0926 20:19:57.885918 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:57.885922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:57.885938 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:19:57.886021 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:57.886023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:57.886079 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:57.886179 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:19:57.886495 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.887001 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.887899 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:57.888137 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:57.888216 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:57.888711 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:19:57.889157 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:19:57.889343 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:57.889464 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 20:19:57.889515 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:19:57.896101 19659 raft_consensus.cc:1261] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a [term 1 FOLLOWER]: Refusing update from remote peer 7a5cac23fd094d5a87c823773e34973c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:57.896149 19790 raft_consensus.cc:1261] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 1 FOLLOWER]: Refusing update from remote peer 7a5cac23fd094d5a87c823773e34973c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:19:57.896775 19990 consensus_queue.cc:1035] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b74f6fc9784f0e8adf93759c13584a" member_type: VOTER last_known_addr { host: "127.31.119.193" port: 35373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:57.896979 19997 consensus_queue.cc:1035] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85d9e96a28c4336aac0d0c495bf9822" member_type: VOTER last_known_addr { host: "127.31.119.194" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:19:57.904799 20001 mvcc.cc:205] Tried to move back new op lower bound from 7333699575374495744 to 7333699575221653504. Current Snapshot: MvccSnapshot[applied={T|T < 7333699575374495744}]
I20260926 20:19:57.905124 20000 mvcc.cc:205] Tried to move back new op lower bound from 7333699575374495744 to 7333699575221653504. Current Snapshot: MvccSnapshot[applied={T|T < 7333699575374495744}]
I20260926 20:19:57.909224 20002 mvcc.cc:205] Tried to move back new op lower bound from 7333699575374495744 to 7333699575221653504. Current Snapshot: MvccSnapshot[applied={T|T < 7333699575374495744}]
I20260926 20:19:57.913928 19517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:57.914145 19517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:19:57.916882 19517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6bb495d65774677a61d265553a8189c: Sending DeleteTablet for 3 replicas of tablet 5fda77eba7244a548c40ce7bd21c63f0
I20260926 20:19:57.917780 19901 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fda77eba7244a548c40ce7bd21c63f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:57 UTC) from {username='slave'} at 127.0.0.1:57586
I20260926 20:19:57.917845 19770 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fda77eba7244a548c40ce7bd21c63f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:57 UTC) from {username='slave'} at 127.0.0.1:34386
I20260926 20:19:57.917938 19639 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fda77eba7244a548c40ce7bd21c63f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:19:57 UTC) from {username='slave'} at 127.0.0.1:44638
I20260926 20:19:57.918416 20012 tablet_replica.cc:330] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a: stopping tablet replica
I20260926 20:19:57.918661 20010 tablet_replica.cc:330] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: stopping tablet replica
I20260926 20:19:57.918797 20012 raft_consensus.cc:2229] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:57.919147 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 19577
I20260926 20:19:57.919167 20012 raft_consensus.cc:2258] T 5fda77eba7244a548c40ce7bd21c63f0 P 39b74f6fc9784f0e8adf93759c13584a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:57.919981 20011 tablet_replica.cc:330] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: stopping tablet replica
I20260926 20:19:57.920274 20011 raft_consensus.cc:2229] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:19:57.920588 20011 raft_consensus.cc:2258] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:57.921243 20010 raft_consensus.cc:2229] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:19:57.921703 20010 raft_consensus.cc:2258] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:19:57.922348 20011 ts_tablet_manager.cc:1905] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:57.923198 20010 ts_tablet_manager.cc:1905] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:19:57.925172 20011 ts_tablet_manager.cc:1918] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:57.925311 20011 log.cc:1196] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-1/wal/wals/5fda77eba7244a548c40ce7bd21c63f0
I20260926 20:19:57.925642 20011 ts_tablet_manager.cc:1939] T 5fda77eba7244a548c40ce7bd21c63f0 P b85d9e96a28c4336aac0d0c495bf9822: Deleting consensus metadata
I20260926 20:19:57.927006 19503 catalog_manager.cc:4841] TS b85d9e96a28c4336aac0d0c495bf9822 (127.31.119.194:37085): tablet 5fda77eba7244a548c40ce7bd21c63f0 (table test-table [id=ad3411c9456548f7aa863e5efaa3d0c9]) successfully deleted
I20260926 20:19:57.927484 20010 ts_tablet_manager.cc:1918] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:19:57.927629 20010 log.cc:1196] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: Deleting WAL directory at /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453737971229-32223-0/minicluster-data/ts-2/wal/wals/5fda77eba7244a548c40ce7bd21c63f0
I20260926 20:19:57.928009 20010 ts_tablet_manager.cc:1939] T 5fda77eba7244a548c40ce7bd21c63f0 P 7a5cac23fd094d5a87c823773e34973c: Deleting consensus metadata
W20260926 20:19:57.928838 19503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.119.193:35373 (error 108)
I20260926 20:19:57.929554 19501 catalog_manager.cc:4841] TS 7a5cac23fd094d5a87c823773e34973c (127.31.119.195:44717): tablet 5fda77eba7244a548c40ce7bd21c63f0 (table test-table [id=ad3411c9456548f7aa863e5efaa3d0c9]) successfully deleted
W20260926 20:19:57.930208 19979 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:19:57.929729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.193:35373 (local address 127.0.0.1:44654)
0926 20:19:57.929865 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:19:57.929883 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:19:57.929981 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:19:57.930081 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.193:35373: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:19:57.930300 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 19708
W20260926 20:19:57.930377 19503 catalog_manager.cc:4584] TS 39b74f6fc9784f0e8adf93759c13584a (127.31.119.193:35373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fda77eba7244a548c40ce7bd21c63f0: Network error: Client connection negotiation failed: client connection to 127.31.119.193:35373: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260926 20:19:57.938838 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 19839
I20260926 20:19:57.945021 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 19486
2026-09-26T20:19:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1295 ms)
[----------] 6 tests from EncryptionPolicyTest (10871 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:19:58 dist-test-slave-vnbh krb5kdc[20016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:19:58 dist-test-slave-vnbh krb5kdc[20016](info): set up 2 sockets
Sep 26 20:19:58 dist-test-slave-vnbh krb5kdc[20016](info): commencing operation
krb5kdc: starting...
W20260926 20:19:58.990531 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:19:59 dist-test-slave-vnbh krb5kdc[20016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:19:59.120118 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:19:59.233393 20029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:19:59.233758 20029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:19:59.233825 20029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:19:59.237922 20029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:19:59.238034 20029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:19:59.238075 20029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:19:59.238111 20029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:19:59.238142 20029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:19:59.243641 20029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:19:59.244913 20029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:19:59.246078 20029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:19:59.256093 20034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:59.256093 20035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:19:59.256125 20037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:19:59.256387 20029 server_base.cc:1034] running on GCE node
I20260926 20:19:59.257071 20029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:19:59.258329 20029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:19:59.259516 20029 hybrid_clock.cc:648] HybridClock initialized: now 1790453999259409 us; error 120 us; skew 500 ppm
Sep 26 20:19:59 dist-test-slave-vnbh krb5kdc[20016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:19:59.267563 20029 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:19:59.268597 20029 webserver.cc:460] Webserver started at http://127.0.0.1:46285/ using document root <none> and password file <none>
I20260926 20:19:59.269237 20029 fs_manager.cc:359] Metadata directory not provided
I20260926 20:19:59.269373 20029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:19:59.269624 20029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:19:59.271775 20029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "fae0119ae7d7435fa3cb4ef036868c3a"
format_stamp: "Formatted at 2026-09-26 20:19:59 on dist-test-slave-vnbh"
server_key: "b4d1b4de1c22ad56133faf712991a9b0"
server_key_iv: "600ecc64d9be9eed5c393cd880ca5306"
server_key_version: "encryptionkey@0"
I20260926 20:19:59.272406 20029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "fae0119ae7d7435fa3cb4ef036868c3a"
format_stamp: "Formatted at 2026-09-26 20:19:59 on dist-test-slave-vnbh"
server_key: "b4d1b4de1c22ad56133faf712991a9b0"
server_key_iv: "600ecc64d9be9eed5c393cd880ca5306"
server_key_version: "encryptionkey@0"
I20260926 20:19:59.276295 20029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 20:19:59.279004 20044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:19:59.280251 20029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:19:59.280402 20029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "fae0119ae7d7435fa3cb4ef036868c3a"
format_stamp: "Formatted at 2026-09-26 20:19:59 on dist-test-slave-vnbh"
server_key: "b4d1b4de1c22ad56133faf712991a9b0"
server_key_iv: "600ecc64d9be9eed5c393cd880ca5306"
server_key_version: "encryptionkey@0"
I20260926 20:19:59.280516 20029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:19:59.292202 20029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:19:59.295727 20029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:19:59.295938 20029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:19:59.304610 20029 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35643
I20260926 20:19:59.304670 20096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35643 every 8 connection(s)
I20260926 20:19:59.305862 20029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:19:59.307114 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 20029
I20260926 20:19:59.307241 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:19:59.307585 32223 external_mini_cluster.cc:1428] Setting key 9efb9ef43608877c3915855b03bb839a
I20260926 20:19:59.309577 20097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:19:59.315266 20097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a: Bootstrap starting.
Sep 26 20:19:59 dist-test-slave-vnbh krb5kdc[20016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:19:59.317950 20097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a: Neither blocks nor log segments found. Creating new log.
I20260926 20:19:59.318710 20097 log.cc:824] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:19:59.320734 20097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a: No bootstrap required, opened a new log
I20260926 20:19:59.322885 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:59.309164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48523 (local address 127.0.0.1:35643)
0926 20:19:59.309707 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:59.309732 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:59.309769 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:19:59.310439 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:59.310457 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:59.310820 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:59.311196 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:59.312035 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:59.313057 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:59.314026 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:59.314547 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:59.314742 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:59.317712 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:59.317718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:59.317732 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:59.317762 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:59.319804 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:59.320362 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:59.320367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:59.320372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:59.320481 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:59.320878 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:59.320880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:59.320881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:59.321269 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:59.321560 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:59.321892 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 20:19:59.322114 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":170,"threads_started":1}
I20260926 20:19:59.324573 20097 raft_consensus.cc:348] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae0119ae7d7435fa3cb4ef036868c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35643 } }
I20260926 20:19:59.324930 20097 raft_consensus.cc:374] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:19:59.325016 20097 raft_consensus.cc:729] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae0119ae7d7435fa3cb4ef036868c3a, State: Initialized, Role: FOLLOWER
I20260926 20:19:59.325542 20097 consensus_queue.cc:260] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae0119ae7d7435fa3cb4ef036868c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35643 } }
I20260926 20:19:59.325734 20097 raft_consensus.cc:388] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:19:59.325794 20097 raft_consensus.cc:482] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:19:59.325887 20097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:19:59.326292 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:19:59.326867 20097 raft_consensus.cc:504] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae0119ae7d7435fa3cb4ef036868c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35643 } }
I20260926 20:19:59.327234 20097 leader_election.cc:302] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fae0119ae7d7435fa3cb4ef036868c3a; no voters:
I20260926 20:19:59.327600 20097 leader_election.cc:288] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:19:59.327674 20103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:19:59.327958 20103 raft_consensus.cc:686] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [term 1 LEADER]: Becoming Leader. State: Replica: fae0119ae7d7435fa3cb4ef036868c3a, State: Running, Role: LEADER
I20260926 20:19:59.328670 20103 consensus_queue.cc:237] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae0119ae7d7435fa3cb4ef036868c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35643 } }
I20260926 20:19:59.329102 20097 sys_catalog.cc:564] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:19:59.330504 20108 sys_catalog.cc:455] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fae0119ae7d7435fa3cb4ef036868c3a. Latest consensus state: current_term: 1 leader_uuid: "fae0119ae7d7435fa3cb4ef036868c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae0119ae7d7435fa3cb4ef036868c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35643 } } }
I20260926 20:19:59.330643 20108 sys_catalog.cc:458] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [sys.catalog]: This master's current role is: LEADER
I20260926 20:19:59.330992 20107 sys_catalog.cc:455] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fae0119ae7d7435fa3cb4ef036868c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae0119ae7d7435fa3cb4ef036868c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35643 } } }
I20260926 20:19:59.331099 20107 sys_catalog.cc:458] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a [sys.catalog]: This master's current role is: LEADER
W20260926 20:19:59.334739 20122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:19:59.334846 20122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:19:59.334949 20117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:19:59.335925 20117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:19:59.339871 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:19:59.328591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59003 (local address 127.0.0.1:35643)
0926 20:19:59.328709 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 20:19:59.328714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:19:59.329044 (+ 330us) server_negotiation.cc:407] Connection header received
0926 20:19:59.329199 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:19:59.329203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:19:59.329260 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:19:59.329380 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:19:59.331534 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:59.332307 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:59.333495 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:19:59.333924 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:19:59.334044 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:19:59.335866 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:19:59.335870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:19:59.335873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:19:59.335900 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:19:59.337739 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:59.338362 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:59.338368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:59.338371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:59.338480 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:19:59.338868 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:19:59.338872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:19:59.338873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:19:59.339107 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:19:59.339264 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 20:19:59.339595 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 20:19:59.339708 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:19:59.342192 20117 catalog_manager.cc:1347] Generated new cluster ID: 6c4be13e1f234180b705625486b31e91
I20260926 20:19:59.342264 20117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:19:59.357720 20117 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:19:59.359068 20117 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:19:59.369825 20117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fae0119ae7d7435fa3cb4ef036868c3a: Generated new TSK 0
I20260926 20:19:59.370573 20117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:19:59.379451 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 20029
W20260926 20:19:59.384893 20014 connection.cc:535] client connection to 127.0.0.1:35643 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35643): Connection reset by peer (error 104)
2026-09-26T20:19:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:19:59 dist-test-slave-vnbh krb5kdc[20128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:19:59 dist-test-slave-vnbh krb5kdc[20128](info): set up 2 sockets
Sep 26 20:19:59 dist-test-slave-vnbh krb5kdc[20128](info): commencing operation
krb5kdc: starting...
W20260926 20:20:00.466828 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:00 dist-test-slave-vnbh krb5kdc[20128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:00.610172 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:00.728557 20141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:00.728870 20141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:00.728915 20141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:00.734370 20141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:00.734453 20141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:00.734475 20141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:00.734494 20141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:00.734510 20141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:00.740281 20141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42103
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:00.741526 20141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:00.742560 20141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:00.751701 20146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:00.751710 20149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:00.751710 20147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:00.751852 20141 server_base.cc:1034] running on GCE node
I20260926 20:20:00.752745 20141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:00.754161 20141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:00.755375 20141 hybrid_clock.cc:648] HybridClock initialized: now 1790454000755341 us; error 53 us; skew 500 ppm
Sep 26 20:20:00 dist-test-slave-vnbh krb5kdc[20128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:00.758597 20141 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:00.759653 20141 webserver.cc:460] Webserver started at http://127.0.0.1:34183/ using document root <none> and password file <none>
I20260926 20:20:00.760242 20141 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:00.760310 20141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:00.760532 20141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:00.762781 20141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "0e50e9f17ad94519a1b6df453ce4af3d"
format_stamp: "Formatted at 2026-09-26 20:20:00 on dist-test-slave-vnbh"
server_key: "cba8edea634fa7c81a4565c1ddd09b4a"
server_key_iv: "faf8b83bdd81c331b41e2febe17a30ae"
server_key_version: "encryptionkey@0"
I20260926 20:20:00.763420 20141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "0e50e9f17ad94519a1b6df453ce4af3d"
format_stamp: "Formatted at 2026-09-26 20:20:00 on dist-test-slave-vnbh"
server_key: "cba8edea634fa7c81a4565c1ddd09b4a"
server_key_iv: "faf8b83bdd81c331b41e2febe17a30ae"
server_key_version: "encryptionkey@0"
I20260926 20:20:00.767690 20141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:00.770453 20156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:00.771924 20141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:00.772105 20141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "0e50e9f17ad94519a1b6df453ce4af3d"
format_stamp: "Formatted at 2026-09-26 20:20:00 on dist-test-slave-vnbh"
server_key: "cba8edea634fa7c81a4565c1ddd09b4a"
server_key_iv: "faf8b83bdd81c331b41e2febe17a30ae"
server_key_version: "encryptionkey@0"
I20260926 20:20:00.772238 20141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:00.798820 20141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:00.802359 20141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:00.802589 20141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:00.810302 20141 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42103
I20260926 20:20:00.810329 20208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42103 every 8 connection(s)
I20260926 20:20:00.811661 20141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:00.814884 20209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:00.818080 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 20141
I20260926 20:20:00.818236 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:00.818542 32223 external_mini_cluster.cc:1428] Setting key e182c7c049658de2306f4febf7fab160
I20260926 20:20:00.821383 20209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d: Bootstrap starting.
I20260926 20:20:00.823638 20209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:00.824409 20209 log.cc:824] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:00.826567 20209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d: No bootstrap required, opened a new log
I20260926 20:20:00.828056 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:00.820072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44955 (local address 127.0.0.1:42103)
0926 20:20:00.820572 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:00.820587 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:00.820623 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:20:00.821339 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:00.821368 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:00.821774 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:00.822125 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:00.822575 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:00.823572 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:00.824547 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:00.825056 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:00.825242 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:00.826246 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:00.826251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:00.826266 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:00.826696 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:00.826773 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:00.826799 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:00.826946 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:00.827134 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 20:20:00.827287 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":164,"threads_started":1}
I20260926 20:20:00.830909 20209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e50e9f17ad94519a1b6df453ce4af3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260926 20:20:00.831184 20209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:00.831256 20209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e50e9f17ad94519a1b6df453ce4af3d, State: Initialized, Role: FOLLOWER
I20260926 20:20:00.831360 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:00.831727 20209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e50e9f17ad94519a1b6df453ce4af3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260926 20:20:00.831892 20209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:00.831977 20209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:00.832075 20209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:00.833086 20209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e50e9f17ad94519a1b6df453ce4af3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260926 20:20:00.833485 20209 leader_election.cc:302] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e50e9f17ad94519a1b6df453ce4af3d; no voters:
I20260926 20:20:00.833879 20209 leader_election.cc:288] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:00.833959 20218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:00.834295 20218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [term 1 LEADER]: Becoming Leader. State: Replica: 0e50e9f17ad94519a1b6df453ce4af3d, State: Running, Role: LEADER
I20260926 20:20:00.834671 20218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e50e9f17ad94519a1b6df453ce4af3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260926 20:20:00.835291 20209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:00.838044 20220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e50e9f17ad94519a1b6df453ce4af3d. Latest consensus state: current_term: 1 leader_uuid: "0e50e9f17ad94519a1b6df453ce4af3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e50e9f17ad94519a1b6df453ce4af3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } } }
I20260926 20:20:00.838196 20220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:00.838442 20221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e50e9f17ad94519a1b6df453ce4af3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e50e9f17ad94519a1b6df453ce4af3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } } }
I20260926 20:20:00.838539 20221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:00.839599 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:00.833920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51575 (local address 127.0.0.1:42103)
0926 20:20:00.834068 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:00.834073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:00.834375 (+ 302us) server_negotiation.cc:407] Connection header received
0926 20:20:00.834775 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:00.834779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:00.834838 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:00.834937 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:00.835223 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:00.836119 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:00.837302 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:00.837689 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:00.837836 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:00.839055 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:00.839063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:00.839069 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:00.839086 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:00.839118 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:00.839133 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:00.839226 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:00.839432 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 20:20:00.839470 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 20:20:00.840749 20233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:00.840939 20233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:00.841022 20235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:00.841828 20235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:00.848366 20235 catalog_manager.cc:1347] Generated new cluster ID: aa8d9adbd3f94f42bc0b1a4c231543ed
I20260926 20:20:00.848476 20235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:00.874933 20235 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:00.876255 20235 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:00.885074 20235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e50e9f17ad94519a1b6df453ce4af3d: Generated new TSK 0
I20260926 20:20:00.885895 20235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:00.890688 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 20141
W20260926 20:20:00.896428 20126 connection.cc:535] client connection to 127.0.0.1:42103 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42103): Connection reset by peer (error 104)
2026-09-26T20:20:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1511 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:00 dist-test-slave-vnbh krb5kdc[20240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:00 dist-test-slave-vnbh krb5kdc[20240](info): set up 2 sockets
Sep 26 20:20:00 dist-test-slave-vnbh krb5kdc[20240](info): commencing operation
krb5kdc: starting...
W20260926 20:20:01.946442 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:02 dist-test-slave-vnbh krb5kdc[20240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:02.073819 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:02.188035 20253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:02.188377 20253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:02.188457 20253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:02.192595 20253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:02.192708 20253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:02.192763 20253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:02.192796 20253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:02.192826 20253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:02.198450 20253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:02.199613 20253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:02.200788 20253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:02.210309 20259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:02.210203 20261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:02.210217 20253 server_base.cc:1034] running on GCE node
W20260926 20:20:02.210203 20258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:02.211145 20253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:02.212546 20253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:02.213814 20253 hybrid_clock.cc:648] HybridClock initialized: now 1790454002213793 us; error 41 us; skew 500 ppm
Sep 26 20:20:02 dist-test-slave-vnbh krb5kdc[20240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:02.217001 20253 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:02.218152 20253 webserver.cc:460] Webserver started at http://127.0.0.1:35819/ using document root <none> and password file <none>
I20260926 20:20:02.218765 20253 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:02.218868 20253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:02.219129 20253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:02.221369 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "2be95439a8294f65a7ad9ceac9d13807"
format_stamp: "Formatted at 2026-09-26 20:20:02 on dist-test-slave-vnbh"
server_key: "b2b9a12e47ba4f72a3c307238345e041"
server_key_iv: "4c3b1da1cdf98767538677f7f510f775"
server_key_version: "encryptionkey@0"
I20260926 20:20:02.221966 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "2be95439a8294f65a7ad9ceac9d13807"
format_stamp: "Formatted at 2026-09-26 20:20:02 on dist-test-slave-vnbh"
server_key: "b2b9a12e47ba4f72a3c307238345e041"
server_key_iv: "4c3b1da1cdf98767538677f7f510f775"
server_key_version: "encryptionkey@0"
I20260926 20:20:02.226071 20253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:20:02.228770 20268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:02.230082 20253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:20:02.230293 20253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "2be95439a8294f65a7ad9ceac9d13807"
format_stamp: "Formatted at 2026-09-26 20:20:02 on dist-test-slave-vnbh"
server_key: "b2b9a12e47ba4f72a3c307238345e041"
server_key_iv: "4c3b1da1cdf98767538677f7f510f775"
server_key_version: "encryptionkey@0"
I20260926 20:20:02.230427 20253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:02.235108 20253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:02.238489 20253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:02.238690 20253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:02.246170 20253 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42421
I20260926 20:20:02.246250 20320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42421 every 8 connection(s)
I20260926 20:20:02.247282 20253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:02.250502 20321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:02.250784 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 20253
I20260926 20:20:02.250936 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:02.251276 32223 external_mini_cluster.cc:1428] Setting key 98938b046d90655889e92d09a96fca6b
I20260926 20:20:02.256891 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:02.252779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60381 (local address 127.0.0.1:42421)
0926 20:20:02.253234 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:02.253254 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:02.253343 (+ 89us) server_negotiation.cc:407] Connection header received
0926 20:20:02.254058 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:02.254076 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:02.254379 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:02.254671 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:02.255610 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:02.255615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:02.255628 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:02.255639 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:02.255696 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:02.255715 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:02.255839 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:02.255952 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 20:20:02.256123 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":159,"threads_started":1}
I20260926 20:20:02.258152 20321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807: Bootstrap starting.
I20260926 20:20:02.260142 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:02.260975 20321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:02.261998 20321 log.cc:824] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:02.264472 20321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807: No bootstrap required, opened a new log
I20260926 20:20:02.264736 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:02.262736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38355 (local address 127.0.0.1:42421)
0926 20:20:02.262970 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:02.262975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:02.263099 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:20:02.263285 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:02.263293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:02.263328 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:02.263415 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:02.264287 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:02.264290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:02.264292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:02.264304 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:02.264330 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:02.264341 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:02.264442 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:02.264548 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:20:02.264596 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260926 20:20:02.268656 20321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be95439a8294f65a7ad9ceac9d13807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } }
I20260926 20:20:02.269033 20321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:02.269122 20321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be95439a8294f65a7ad9ceac9d13807, State: Initialized, Role: FOLLOWER
I20260926 20:20:02.269713 20321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be95439a8294f65a7ad9ceac9d13807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } }
I20260926 20:20:02.269882 20321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:02.269951 20321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:02.270054 20321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:02.271044 20321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be95439a8294f65a7ad9ceac9d13807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } }
I20260926 20:20:02.271435 20321 leader_election.cc:302] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2be95439a8294f65a7ad9ceac9d13807; no voters:
I20260926 20:20:02.271842 20321 leader_election.cc:288] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:02.272037 20331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:02.272383 20331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [term 1 LEADER]: Becoming Leader. State: Replica: 2be95439a8294f65a7ad9ceac9d13807, State: Running, Role: LEADER
I20260926 20:20:02.272706 20331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be95439a8294f65a7ad9ceac9d13807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } }
I20260926 20:20:02.273262 20321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:02.274523 20333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2be95439a8294f65a7ad9ceac9d13807. Latest consensus state: current_term: 1 leader_uuid: "2be95439a8294f65a7ad9ceac9d13807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be95439a8294f65a7ad9ceac9d13807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } } }
I20260926 20:20:02.274667 20333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:02.275094 20332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2be95439a8294f65a7ad9ceac9d13807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be95439a8294f65a7ad9ceac9d13807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42421 } } }
I20260926 20:20:02.275141 20340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:02.275223 20332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:02.278358 20340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:02.284503 20340 catalog_manager.cc:1347] Generated new cluster ID: a83b6ee8d05d46f89e6f78f6a2905ed4
I20260926 20:20:02.284595 20340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:02.297019 20340 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:02.297427 20340 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:02.304128 20340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2be95439a8294f65a7ad9ceac9d13807: Generated new TSK 0
I20260926 20:20:02.304994 20340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:02.318462 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 20253
2026-09-26T20:20:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:02 dist-test-slave-vnbh krb5kdc[20352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:02 dist-test-slave-vnbh krb5kdc[20352](info): set up 2 sockets
Sep 26 20:20:02 dist-test-slave-vnbh krb5kdc[20352](info): commencing operation
krb5kdc: starting...
W20260926 20:20:03.394429 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:03 dist-test-slave-vnbh krb5kdc[20352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:03.525190 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:03.642788 20365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:03.643175 20365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:03.643267 20365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:03.647557 20365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:03.647673 20365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:03.647715 20365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:03.647749 20365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:03.647780 20365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:03.653453 20365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:03.654723 20365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:03.655912 20365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:03.665467 20370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:03.665473 20373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:03.665460 20371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:03.665470 20365 server_base.cc:1034] running on GCE node
I20260926 20:20:03.666471 20365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:03.667747 20365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:03.668979 20365 hybrid_clock.cc:648] HybridClock initialized: now 1790454003668962 us; error 38 us; skew 500 ppm
Sep 26 20:20:03 dist-test-slave-vnbh krb5kdc[20352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:03.672340 20365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:03.673485 20365 webserver.cc:460] Webserver started at http://127.0.0.1:40661/ using document root <none> and password file <none>
I20260926 20:20:03.674098 20365 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:03.674204 20365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:03.674450 20365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:03.676630 20365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "cbabde8d466b475d81fa6671dbda3379"
format_stamp: "Formatted at 2026-09-26 20:20:03 on dist-test-slave-vnbh"
server_key: "cfde099515ad1a61ed67ee58140a1571"
server_key_iv: "778aa111144f6cc1009d5bb4a4a758ff"
server_key_version: "encryptionkey@0"
I20260926 20:20:03.677251 20365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "cbabde8d466b475d81fa6671dbda3379"
format_stamp: "Formatted at 2026-09-26 20:20:03 on dist-test-slave-vnbh"
server_key: "cfde099515ad1a61ed67ee58140a1571"
server_key_iv: "778aa111144f6cc1009d5bb4a4a758ff"
server_key_version: "encryptionkey@0"
I20260926 20:20:03.681236 20365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:20:03.684018 20380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:03.685526 20365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:20:03.685688 20365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "cbabde8d466b475d81fa6671dbda3379"
format_stamp: "Formatted at 2026-09-26 20:20:03 on dist-test-slave-vnbh"
server_key: "cfde099515ad1a61ed67ee58140a1571"
server_key_iv: "778aa111144f6cc1009d5bb4a4a758ff"
server_key_version: "encryptionkey@0"
I20260926 20:20:03.685822 20365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:03.700423 20365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:03.704021 20365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:03.704238 20365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:03.712253 20365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42745
I20260926 20:20:03.712273 20432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42745 every 8 connection(s)
I20260926 20:20:03.713478 20365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:03.716449 20433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:03.722316 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 20365
I20260926 20:20:03.722414 20433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379: Bootstrap starting.
I20260926 20:20:03.722496 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:03.722824 32223 external_mini_cluster.cc:1428] Setting key e5f423bf3f87304bc74dc4723e203f5b
I20260926 20:20:03.725512 20433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:03.726423 20433 log.cc:824] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:03.728823 20433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379: No bootstrap required, opened a new log
Sep 26 20:20:03 dist-test-slave-vnbh krb5kdc[20352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:20:03.733120 20433 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbabde8d466b475d81fa6671dbda3379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } }
I20260926 20:20:03.733443 20433 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:03.733541 20433 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbabde8d466b475d81fa6671dbda3379, State: Initialized, Role: FOLLOWER
I20260926 20:20:03.734046 20433 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbabde8d466b475d81fa6671dbda3379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } }
I20260926 20:20:03.734242 20433 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:03.734313 20433 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:03.734433 20433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:03.735793 20433 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbabde8d466b475d81fa6671dbda3379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } }
I20260926 20:20:03.736204 20433 leader_election.cc:302] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbabde8d466b475d81fa6671dbda3379; no voters:
I20260926 20:20:03.736629 20433 leader_election.cc:288] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:03.736711 20438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:03.736935 20438 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [term 1 LEADER]: Becoming Leader. State: Replica: cbabde8d466b475d81fa6671dbda3379, State: Running, Role: LEADER
I20260926 20:20:03.737352 20438 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbabde8d466b475d81fa6671dbda3379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } }
I20260926 20:20:03.738135 20433 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:03.739236 20439 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbabde8d466b475d81fa6671dbda3379. Latest consensus state: current_term: 1 leader_uuid: "cbabde8d466b475d81fa6671dbda3379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbabde8d466b475d81fa6671dbda3379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } } }
I20260926 20:20:03.739725 20439 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:03.739619 20440 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbabde8d466b475d81fa6671dbda3379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbabde8d466b475d81fa6671dbda3379" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42745 } } }
I20260926 20:20:03.740042 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:03.724320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35033 (local address 127.0.0.1:42745)
0926 20:20:03.724834 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:03.724853 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:03.725047 (+ 194us) server_negotiation.cc:407] Connection header received
0926 20:20:03.726033 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:03.726056 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:03.726518 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:03.726899 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:03.727726 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:03.729325 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:03.730216 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:03.730943 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:03.731110 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:03.734014 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:03.734020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:03.734036 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:03.734072 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:03.736212 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:03.736780 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:03.736789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:03.736797 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:03.736975 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:03.737391 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:03.737394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:03.737395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:03.737867 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:03.738094 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:03.738403 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 20:20:03.738762 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":154,"threads_started":1}
I20260926 20:20:03.740087 20440 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:03.743606 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 20:20:03.743968 20453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:03.744058 20453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:03.744140 20449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:03.745062 20449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:03.752534 20449 catalog_manager.cc:1347] Generated new cluster ID: 4e4c6803f4fd42d3b364619013d468e5
I20260926 20:20:03.752610 20449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:03.755012 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:03.745868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57729 (local address 127.0.0.1:42745)
0926 20:20:03.745986 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:03.745991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:03.746309 (+ 318us) server_negotiation.cc:407] Connection header received
0926 20:20:03.746450 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:03.746453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:03.746499 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:03.746592 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:03.747371 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:03.748117 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:03.749233 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:03.749560 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:03.749625 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:03.751346 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:03.751350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:03.751353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:03.751379 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:03.753151 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:03.753721 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:03.753725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:03.753727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:03.753815 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:03.754130 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:03.754133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:03.754134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:03.754335 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:03.754448 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:03.754729 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 20:20:03.754849 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:20:03.764673 20449 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:03.766115 20449 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:03.777081 20449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbabde8d466b475d81fa6671dbda3379: Generated new TSK 0
I20260926 20:20:03.777884 20449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:03.787132 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 20365
2026-09-26T20:20:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:03 dist-test-slave-vnbh krb5kdc[20464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:03 dist-test-slave-vnbh krb5kdc[20464](info): set up 2 sockets
Sep 26 20:20:03 dist-test-slave-vnbh krb5kdc[20464](info): commencing operation
krb5kdc: starting...
W20260926 20:20:04.894810 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:04 dist-test-slave-vnbh krb5kdc[20464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:05.025233 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:05.139706 20477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:05.140009 20477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:05.140059 20477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:05.144068 20477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:05.144131 20477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:05.144150 20477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:05.144168 20477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:05.144183 20477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:05.149591 20477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:05.150667 20477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:05.151695 20477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:05.160634 20482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:05.160643 20485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:05.160845 20477 server_base.cc:1034] running on GCE node
W20260926 20:20:05.160643 20483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:05.161635 20477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:05.162835 20477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:05.164047 20477 hybrid_clock.cc:648] HybridClock initialized: now 1790454005164020 us; error 62 us; skew 500 ppm
Sep 26 20:20:05 dist-test-slave-vnbh krb5kdc[20464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:05.170892 20477 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:05.172008 20477 webserver.cc:460] Webserver started at http://127.0.0.1:41459/ using document root <none> and password file <none>
I20260926 20:20:05.172678 20477 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:05.172783 20477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:05.173029 20477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:05.175390 20477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "11f383ddc2134397860cabe1f5bc9ab5"
format_stamp: "Formatted at 2026-09-26 20:20:05 on dist-test-slave-vnbh"
server_key: "121760ed5080e9e091e058f35e5a9ee5"
server_key_iv: "a905d30fe2d772d2abb7a16eaea15098"
server_key_version: "encryptionkey@0"
I20260926 20:20:05.176038 20477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "11f383ddc2134397860cabe1f5bc9ab5"
format_stamp: "Formatted at 2026-09-26 20:20:05 on dist-test-slave-vnbh"
server_key: "121760ed5080e9e091e058f35e5a9ee5"
server_key_iv: "a905d30fe2d772d2abb7a16eaea15098"
server_key_version: "encryptionkey@0"
I20260926 20:20:05.180052 20477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:05.182638 20492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:05.183991 20477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:20:05.184126 20477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "11f383ddc2134397860cabe1f5bc9ab5"
format_stamp: "Formatted at 2026-09-26 20:20:05 on dist-test-slave-vnbh"
server_key: "121760ed5080e9e091e058f35e5a9ee5"
server_key_iv: "a905d30fe2d772d2abb7a16eaea15098"
server_key_version: "encryptionkey@0"
I20260926 20:20:05.184237 20477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:05.194212 20477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:05.197707 20477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:05.197969 20477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:05.205508 20477 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44975
I20260926 20:20:05.205529 20544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44975 every 8 connection(s)
I20260926 20:20:05.206611 20477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:05.209580 20545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:05.212373 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 20477
I20260926 20:20:05.212522 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:05.212844 32223 external_mini_cluster.cc:1428] Setting key 383d4ac77aaac3cabbca72d97470b4cf
I20260926 20:20:05.216037 20545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5: Bootstrap starting.
I20260926 20:20:05.218420 20545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:05.219622 20545 log.cc:824] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:05.222059 20545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5: No bootstrap required, opened a new log
I20260926 20:20:05.222579 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:05.214371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60205 (local address 127.0.0.1:44975)
0926 20:20:05.214879 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:05.214895 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:05.214933 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:20:05.215610 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:05.215630 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:05.216006 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:05.216402 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:05.216858 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:05.217960 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:05.218999 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:05.219520 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:05.219725 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:05.220682 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:05.220688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:05.220703 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:05.221115 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:05.221173 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:05.221196 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:05.221342 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:05.221531 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:20:05.221761 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":140,"threads_started":1}
I20260926 20:20:05.225613 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:05.226161 20545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f383ddc2134397860cabe1f5bc9ab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44975 } }
I20260926 20:20:05.226430 20545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:05.226516 20545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f383ddc2134397860cabe1f5bc9ab5, State: Initialized, Role: FOLLOWER
I20260926 20:20:05.227020 20545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f383ddc2134397860cabe1f5bc9ab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44975 } }
I20260926 20:20:05.227181 20545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:05.227272 20545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:05.227397 20545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:05.228788 20545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f383ddc2134397860cabe1f5bc9ab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44975 } }
I20260926 20:20:05.229182 20545 leader_election.cc:302] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11f383ddc2134397860cabe1f5bc9ab5; no voters:
I20260926 20:20:05.229555 20545 leader_election.cc:288] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:05.229668 20555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:05.229954 20555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [term 1 LEADER]: Becoming Leader. State: Replica: 11f383ddc2134397860cabe1f5bc9ab5, State: Running, Role: LEADER
I20260926 20:20:05.230269 20555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f383ddc2134397860cabe1f5bc9ab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44975 } }
I20260926 20:20:05.230997 20545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:05.231673 20556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11f383ddc2134397860cabe1f5bc9ab5. Latest consensus state: current_term: 1 leader_uuid: "11f383ddc2134397860cabe1f5bc9ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f383ddc2134397860cabe1f5bc9ab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44975 } } }
I20260926 20:20:05.232420 20556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:05.233011 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:05.227926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44411 (local address 127.0.0.1:44975)
0926 20:20:05.228082 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:05.228086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:05.228234 (+ 148us) server_negotiation.cc:407] Connection header received
0926 20:20:05.228329 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:05.228331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:05.228400 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:05.228482 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:05.228738 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:05.229548 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:05.230728 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:05.231086 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:05.231188 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:05.232172 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:05.232176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:05.232179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:05.232198 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:05.232225 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:05.232235 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:05.232329 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:05.232742 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 20:20:05.232830 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:20:05.233048 20557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11f383ddc2134397860cabe1f5bc9ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f383ddc2134397860cabe1f5bc9ab5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44975 } } }
I20260926 20:20:05.233675 20557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:05.234328 20565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:05.237627 20565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:05.243546 20565 catalog_manager.cc:1347] Generated new cluster ID: 8eb21a6abc2640ccb06958ec2363f61d
I20260926 20:20:05.243625 20565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:05.264485 20565 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:05.265672 20565 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:05.277935 20565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11f383ddc2134397860cabe1f5bc9ab5: Generated new TSK 0
I20260926 20:20:05.278734 20565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:05.291172 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 20477
2026-09-26T20:20:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:05 dist-test-slave-vnbh krb5kdc[20576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:05 dist-test-slave-vnbh krb5kdc[20576](info): set up 2 sockets
Sep 26 20:20:05 dist-test-slave-vnbh krb5kdc[20576](info): commencing operation
krb5kdc: starting...
W20260926 20:20:06.362599 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:06 dist-test-slave-vnbh krb5kdc[20576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:06.493566 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:06.610908 20589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:06.611210 20589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:06.611253 20589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:06.615535 20589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:06.615621 20589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:06.615641 20589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:06.615659 20589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:06.615675 20589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:06.621595 20589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:06.622813 20589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:06.623955 20589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:06.633507 20594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:06.633526 20597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:06.633520 20595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:06.633764 20589 server_base.cc:1034] running on GCE node
I20260926 20:20:06.634579 20589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:06.635792 20589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:06.637012 20589 hybrid_clock.cc:648] HybridClock initialized: now 1790454006636982 us; error 45 us; skew 500 ppm
Sep 26 20:20:06 dist-test-slave-vnbh krb5kdc[20576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:06.640333 20589 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:06.641388 20589 webserver.cc:460] Webserver started at http://127.0.0.1:39713/ using document root <none> and password file <none>
I20260926 20:20:06.641994 20589 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:06.642058 20589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:06.642239 20589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:06.644434 20589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "a347fe53ae21430cb515b7cca07aceda"
format_stamp: "Formatted at 2026-09-26 20:20:06 on dist-test-slave-vnbh"
server_key: "35b5c81d06dabc99a55b9fab9ddcf407"
server_key_iv: "cbef45ea87427440f750a70fba665b3a"
server_key_version: "encryptionkey@0"
I20260926 20:20:06.645022 20589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "a347fe53ae21430cb515b7cca07aceda"
format_stamp: "Formatted at 2026-09-26 20:20:06 on dist-test-slave-vnbh"
server_key: "35b5c81d06dabc99a55b9fab9ddcf407"
server_key_iv: "cbef45ea87427440f750a70fba665b3a"
server_key_version: "encryptionkey@0"
I20260926 20:20:06.649142 20589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:20:06.651962 20604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:06.653602 20589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:06.653741 20589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "a347fe53ae21430cb515b7cca07aceda"
format_stamp: "Formatted at 2026-09-26 20:20:06 on dist-test-slave-vnbh"
server_key: "35b5c81d06dabc99a55b9fab9ddcf407"
server_key_iv: "cbef45ea87427440f750a70fba665b3a"
server_key_version: "encryptionkey@0"
I20260926 20:20:06.653848 20589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:06.658859 20589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:06.662343 20589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:06.662499 20589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:06.669693 20589 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38505
I20260926 20:20:06.669720 20656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38505 every 8 connection(s)
I20260926 20:20:06.670694 20589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:06.673820 20657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:06.679817 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda: Bootstrap starting.
I20260926 20:20:06.680650 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 20589
I20260926 20:20:06.680799 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:06.681095 32223 external_mini_cluster.cc:1428] Setting key 1f9fe2372cf096b38f71b581b7f6de2d
I20260926 20:20:06.683018 20657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:06.683784 20657 log.cc:824] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:06.686213 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda: No bootstrap required, opened a new log
I20260926 20:20:06.686789 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:06.682461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35087 (local address 127.0.0.1:38505)
0926 20:20:06.682943 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:06.682958 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:06.682993 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:20:06.683713 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:06.683731 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:06.684091 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:06.684482 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:06.685428 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:06.685433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:06.685444 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:06.685454 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:06.685508 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:06.685526 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:06.685616 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:06.685799 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 20:20:06.686000 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":145,"threads_started":1}
I20260926 20:20:06.690018 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:06.690611 20657 raft_consensus.cc:348] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347fe53ae21430cb515b7cca07aceda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38505 } }
I20260926 20:20:06.690908 20657 raft_consensus.cc:374] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:06.690971 20657 raft_consensus.cc:729] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a347fe53ae21430cb515b7cca07aceda, State: Initialized, Role: FOLLOWER
I20260926 20:20:06.691475 20657 consensus_queue.cc:260] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347fe53ae21430cb515b7cca07aceda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38505 } }
I20260926 20:20:06.691663 20657 raft_consensus.cc:388] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:06.691737 20657 raft_consensus.cc:482] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:06.691841 20657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:06.693179 20657 raft_consensus.cc:504] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347fe53ae21430cb515b7cca07aceda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38505 } }
I20260926 20:20:06.693631 20657 leader_election.cc:302] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a347fe53ae21430cb515b7cca07aceda; no voters:
I20260926 20:20:06.694036 20657 leader_election.cc:288] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:06.694281 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:06.692387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43377 (local address 127.0.0.1:38505)
0926 20:20:06.692527 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:06.692531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:06.692716 (+ 185us) server_negotiation.cc:407] Connection header received
0926 20:20:06.692874 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:06.692876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:06.692906 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:06.692970 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:06.693827 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:06.693831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:06.693836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:06.693851 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:06.693879 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:06.693892 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:06.694015 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:06.694086 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:20:06.694142 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:20:06.694304 20667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:06.694691 20667 raft_consensus.cc:686] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [term 1 LEADER]: Becoming Leader. State: Replica: a347fe53ae21430cb515b7cca07aceda, State: Running, Role: LEADER
I20260926 20:20:06.695016 20667 consensus_queue.cc:237] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347fe53ae21430cb515b7cca07aceda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38505 } }
I20260926 20:20:06.695366 20657 sys_catalog.cc:564] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:06.696190 20667 sys_catalog.cc:455] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [sys.catalog]: SysCatalogTable state changed. Reason: New leader a347fe53ae21430cb515b7cca07aceda. Latest consensus state: current_term: 1 leader_uuid: "a347fe53ae21430cb515b7cca07aceda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347fe53ae21430cb515b7cca07aceda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38505 } } }
I20260926 20:20:06.696354 20667 sys_catalog.cc:458] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:06.696733 20668 sys_catalog.cc:455] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a347fe53ae21430cb515b7cca07aceda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347fe53ae21430cb515b7cca07aceda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38505 } } }
I20260926 20:20:06.696890 20668 sys_catalog.cc:458] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:06.697252 20676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:06.701340 20676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:06.707155 20676 catalog_manager.cc:1347] Generated new cluster ID: bac1a1696a334787a808442f85ba0257
I20260926 20:20:06.707234 20676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:06.721612 20676 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:06.721973 20676 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:06.728577 20676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a347fe53ae21430cb515b7cca07aceda: Generated new TSK 0
I20260926 20:20:06.729580 20676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:06.738133 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 20589
2026-09-26T20:20:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:06 dist-test-slave-vnbh krb5kdc[20688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:06 dist-test-slave-vnbh krb5kdc[20688](info): set up 2 sockets
Sep 26 20:20:06 dist-test-slave-vnbh krb5kdc[20688](info): commencing operation
krb5kdc: starting...
W20260926 20:20:07.790455 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:07 dist-test-slave-vnbh krb5kdc[20688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:07.920991 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:08.038368 20701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:08.038721 20701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:08.038802 20701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:08.043111 20701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:08.043220 20701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:08.043257 20701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:08.043299 20701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:08.043331 20701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:08.049037 20701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:08.050341 20701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:08.051477 20701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:08.061228 20709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:08.061189 20707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:08.061232 20701 server_base.cc:1034] running on GCE node
W20260926 20:20:08.061527 20706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:08.062206 20701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:08.063503 20701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:08.064721 20701 hybrid_clock.cc:648] HybridClock initialized: now 1790454008064695 us; error 43 us; skew 500 ppm
Sep 26 20:20:08 dist-test-slave-vnbh krb5kdc[20688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:08.067843 20701 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:08.068934 20701 webserver.cc:460] Webserver started at http://127.0.0.1:33643/ using document root <none> and password file <none>
I20260926 20:20:08.069582 20701 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:08.069653 20701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:08.069834 20701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:08.072115 20701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "26836960b6f042b7ba9d1facf7d75c44"
format_stamp: "Formatted at 2026-09-26 20:20:08 on dist-test-slave-vnbh"
server_key: "9791604b3de7a8c56cab47ad5654734a"
server_key_iv: "30c91f06639ac55bd1fc5052b4305756"
server_key_version: "encryptionkey@0"
I20260926 20:20:08.072693 20701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "26836960b6f042b7ba9d1facf7d75c44"
format_stamp: "Formatted at 2026-09-26 20:20:08 on dist-test-slave-vnbh"
server_key: "9791604b3de7a8c56cab47ad5654734a"
server_key_iv: "30c91f06639ac55bd1fc5052b4305756"
server_key_version: "encryptionkey@0"
I20260926 20:20:08.077667 20701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 20:20:08.080860 20716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:08.082764 20701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:08.082958 20701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "26836960b6f042b7ba9d1facf7d75c44"
format_stamp: "Formatted at 2026-09-26 20:20:08 on dist-test-slave-vnbh"
server_key: "9791604b3de7a8c56cab47ad5654734a"
server_key_iv: "30c91f06639ac55bd1fc5052b4305756"
server_key_version: "encryptionkey@0"
I20260926 20:20:08.083144 20701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:08.106509 20701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:08.110178 20701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:08.110420 20701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:08.118077 20701 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42873
I20260926 20:20:08.118081 20768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42873 every 8 connection(s)
I20260926 20:20:08.119167 20701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:08.122349 20769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:08.128291 20769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44: Bootstrap starting.
I20260926 20:20:08.128746 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 20701
I20260926 20:20:08.128908 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:08.129225 32223 external_mini_cluster.cc:1428] Setting key bdbb4a6117cd82ef46816d877c7e5960
I20260926 20:20:08.131340 20769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:08.132135 20769 log.cc:824] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:08.134492 20769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44: No bootstrap required, opened a new log
I20260926 20:20:08.138603 20769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26836960b6f042b7ba9d1facf7d75c44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } }
I20260926 20:20:08.138921 20769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:08.138988 20769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26836960b6f042b7ba9d1facf7d75c44, State: Initialized, Role: FOLLOWER
I20260926 20:20:08.139619 20769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26836960b6f042b7ba9d1facf7d75c44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } }
I20260926 20:20:08.139838 20769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:08.139914 20769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:08.140010 20769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 20:20:08 dist-test-slave-vnbh krb5kdc[20688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:20:08.141129 20769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26836960b6f042b7ba9d1facf7d75c44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } }
I20260926 20:20:08.141577 20769 leader_election.cc:302] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26836960b6f042b7ba9d1facf7d75c44; no voters:
I20260926 20:20:08.141987 20769 leader_election.cc:288] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:08.142370 20774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:08.142594 20774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [term 1 LEADER]: Becoming Leader. State: Replica: 26836960b6f042b7ba9d1facf7d75c44, State: Running, Role: LEADER
I20260926 20:20:08.143265 20774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26836960b6f042b7ba9d1facf7d75c44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } }
I20260926 20:20:08.143457 20769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:08.145071 20776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26836960b6f042b7ba9d1facf7d75c44. Latest consensus state: current_term: 1 leader_uuid: "26836960b6f042b7ba9d1facf7d75c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26836960b6f042b7ba9d1facf7d75c44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } } }
I20260926 20:20:08.145256 20776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:08.145613 20775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26836960b6f042b7ba9d1facf7d75c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26836960b6f042b7ba9d1facf7d75c44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } } }
I20260926 20:20:08.145707 20775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:08.148218 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:08.130762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:41817 (local address 127.0.0.1:42873)
0926 20:20:08.131294 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:08.131310 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:08.131349 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:20:08.132136 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:08.132156 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:08.132665 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:08.133081 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:08.134377 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:08.135825 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:08.136760 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:08.138169 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:08.138356 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:08.141503 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:08.141510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:08.141530 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:08.141568 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:08.143979 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:08.144761 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:08.144766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:08.144770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:08.144889 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:08.145323 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:08.145327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:08.145329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:08.145879 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:08.146106 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:08.146949 (+ 843us) server_negotiation.cc:299] Negotiation successful
0926 20:20:08.147201 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":148,"threads_started":1}
W20260926 20:20:08.148712 20789 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:08.148808 20789 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:08.148892 20788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:08.149816 20788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:08.151806 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:08.155975 20788 catalog_manager.cc:1347] Generated new cluster ID: 81000c5fa6bd476c9ab985b1f31ceea7
I20260926 20:20:08.156059 20788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:08.164004 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:08.154140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:53055 (local address 127.0.0.1:42873)
0926 20:20:08.154250 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:08.154254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:08.154498 (+ 244us) server_negotiation.cc:407] Connection header received
0926 20:20:08.154683 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:08.154687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:08.154754 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:08.154808 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:08.155786 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:08.156743 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:08.158004 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:08.158405 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:08.158505 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:08.160275 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:08.160279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:08.160282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:08.160307 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:08.162035 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:08.162598 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:08.162602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:08.162604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:08.162691 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:08.163042 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:08.163045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:08.163046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:08.163267 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:08.163440 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:08.163734 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 20:20:08.163841 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:20:08.164990 20788 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:08.165993 20788 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:08.174324 20788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26836960b6f042b7ba9d1facf7d75c44: Generated new TSK 0
I20260926 20:20:08.175361 20788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:08.186679 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 20701
W20260926 20:20:08.192479 20686 connection.cc:535] client connection to 127.0.0.1:42873 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42873): Connection reset by peer (error 104)
2026-09-26T20:20:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:08 dist-test-slave-vnbh krb5kdc[20800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:08 dist-test-slave-vnbh krb5kdc[20800](info): set up 2 sockets
Sep 26 20:20:08 dist-test-slave-vnbh krb5kdc[20800](info): commencing operation
krb5kdc: starting...
W20260926 20:20:09.230520 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:09 dist-test-slave-vnbh krb5kdc[20800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:09.361721 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:09.476727 20813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:09.477039 20813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:09.477083 20813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:09.481415 20813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:09.481503 20813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:09.481523 20813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:09.481540 20813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:09.481555 20813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:09.487511 20813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:09.488708 20813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:09.489760 20813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:09.498641 20819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:09.498641 20818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:09.498674 20821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:09.499094 20813 server_base.cc:1034] running on GCE node
I20260926 20:20:09.499688 20813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:09.501016 20813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:09.502229 20813 hybrid_clock.cc:648] HybridClock initialized: now 1790454009502217 us; error 61 us; skew 500 ppm
Sep 26 20:20:09 dist-test-slave-vnbh krb5kdc[20800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:09.505203 20813 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:09.506156 20813 webserver.cc:460] Webserver started at http://127.0.0.1:33587/ using document root <none> and password file <none>
I20260926 20:20:09.506692 20813 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:09.506754 20813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:09.506922 20813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:09.509064 20813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "465d7b2a09ef44748619a655fbcf6165"
format_stamp: "Formatted at 2026-09-26 20:20:09 on dist-test-slave-vnbh"
server_key: "84d8364e2e9456fe10f0bcc639c286ee"
server_key_iv: "a8b397608be869f7cf38c836232c4faf"
server_key_version: "encryptionkey@0"
I20260926 20:20:09.509702 20813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "465d7b2a09ef44748619a655fbcf6165"
format_stamp: "Formatted at 2026-09-26 20:20:09 on dist-test-slave-vnbh"
server_key: "84d8364e2e9456fe10f0bcc639c286ee"
server_key_iv: "a8b397608be869f7cf38c836232c4faf"
server_key_version: "encryptionkey@0"
I20260926 20:20:09.513600 20813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:20:09.516258 20828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:09.517794 20813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:09.517966 20813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "465d7b2a09ef44748619a655fbcf6165"
format_stamp: "Formatted at 2026-09-26 20:20:09 on dist-test-slave-vnbh"
server_key: "84d8364e2e9456fe10f0bcc639c286ee"
server_key_iv: "a8b397608be869f7cf38c836232c4faf"
server_key_version: "encryptionkey@0"
I20260926 20:20:09.518074 20813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:09.541415 20813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:09.545364 20813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:09.545547 20813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:09.553779 20813 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37689
I20260926 20:20:09.553740 20880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37689 every 8 connection(s)
I20260926 20:20:09.554841 20813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:09.557988 20881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:09.559137 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 20813
I20260926 20:20:09.559312 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:09.559629 32223 external_mini_cluster.cc:1428] Setting key aef21c6404be7cd43ada96ec13e8acc4
I20260926 20:20:09.564543 20881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165: Bootstrap starting.
I20260926 20:20:09.567094 20881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:09.567899 20881 log.cc:824] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:09.569203 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:09.561233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:60667 (local address 127.0.0.1:37689)
0926 20:20:09.561717 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:09.561732 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:09.561775 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:20:09.562429 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:09.562447 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:09.562814 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:09.563159 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:09.563587 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:09.564581 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:09.565644 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:09.566126 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:09.566316 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:09.567289 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:09.567294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:09.567307 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:09.567696 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:09.567753 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:09.567784 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:09.567937 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:09.568117 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 20:20:09.568269 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":141,"threads_started":1}
I20260926 20:20:09.570397 20881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165: No bootstrap required, opened a new log
I20260926 20:20:09.572728 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:09.574659 20881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d7b2a09ef44748619a655fbcf6165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37689 } }
I20260926 20:20:09.575055 20881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:09.575130 20881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465d7b2a09ef44748619a655fbcf6165, State: Initialized, Role: FOLLOWER
I20260926 20:20:09.575634 20881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d7b2a09ef44748619a655fbcf6165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37689 } }
I20260926 20:20:09.575798 20881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:09.575873 20881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:09.575999 20881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:09.577004 20881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d7b2a09ef44748619a655fbcf6165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37689 } }
I20260926 20:20:09.577437 20881 leader_election.cc:302] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 465d7b2a09ef44748619a655fbcf6165; no voters:
I20260926 20:20:09.577795 20881 leader_election.cc:288] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:09.577867 20891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:09.578078 20891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [term 1 LEADER]: Becoming Leader. State: Replica: 465d7b2a09ef44748619a655fbcf6165, State: Running, Role: LEADER
I20260926 20:20:09.578474 20891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d7b2a09ef44748619a655fbcf6165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37689 } }
I20260926 20:20:09.579197 20881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:09.579628 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:09.574868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:47917 (local address 127.0.0.1:37689)
0926 20:20:09.575110 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:09.575114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:09.575384 (+ 270us) server_negotiation.cc:407] Connection header received
0926 20:20:09.575524 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:09.575527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:09.575579 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:09.575700 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:09.575974 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:09.576536 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:09.577501 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:09.577818 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:09.577937 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:09.579117 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:09.579120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:09.579123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:09.579136 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:09.579166 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:09.579177 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:09.579283 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:09.579449 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:20:09.579485 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260926 20:20:09.579873 20891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 465d7b2a09ef44748619a655fbcf6165. Latest consensus state: current_term: 1 leader_uuid: "465d7b2a09ef44748619a655fbcf6165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d7b2a09ef44748619a655fbcf6165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37689 } } }
I20260926 20:20:09.580006 20891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:09.580585 20892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "465d7b2a09ef44748619a655fbcf6165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465d7b2a09ef44748619a655fbcf6165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37689 } } }
I20260926 20:20:09.580912 20892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165 [sys.catalog]: This master's current role is: LEADER
W20260926 20:20:09.584134 20906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:09.584214 20906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:09.584299 20900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:09.585074 20900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:09.591179 20900 catalog_manager.cc:1347] Generated new cluster ID: 236b91dd52d84e63a3f38bd533933e8f
I20260926 20:20:09.591269 20900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:09.606913 20900 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:09.608117 20900 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:09.620464 20900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 465d7b2a09ef44748619a655fbcf6165: Generated new TSK 0
I20260926 20:20:09.621487 20900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:09.634052 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 20813
W20260926 20:20:09.639598 20798 connection.cc:535] client connection to 127.0.0.1:37689 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37689): Connection reset by peer (error 104)
2026-09-26T20:20:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:09 dist-test-slave-vnbh krb5kdc[20912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:09 dist-test-slave-vnbh krb5kdc[20912](info): set up 2 sockets
Sep 26 20:20:09 dist-test-slave-vnbh krb5kdc[20912](info): commencing operation
krb5kdc: starting...
W20260926 20:20:10.714497 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:10 dist-test-slave-vnbh krb5kdc[20912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:10.845654 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:10.960239 20925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:10.960588 20925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:10.960672 20925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:10.964963 20925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:10.965080 20925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:10.965119 20925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:10.965158 20925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:10.965193 20925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:10.970872 20925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:10.972118 20925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:10.973246 20925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:20:10.982654 20925 server_base.cc:1034] running on GCE node
W20260926 20:20:10.982671 20931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:10.982656 20933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:10.982985 20930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:10.983660 20925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:10.984943 20925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:10.986121 20925 hybrid_clock.cc:648] HybridClock initialized: now 1790454010986066 us; error 69 us; skew 500 ppm
Sep 26 20:20:10 dist-test-slave-vnbh krb5kdc[20912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:10.995610 20925 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:10.996815 20925 webserver.cc:460] Webserver started at http://127.0.0.1:45605/ using document root <none> and password file <none>
I20260926 20:20:10.997540 20925 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:10.997638 20925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:10.997887 20925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:11.000172 20925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "0aab19aead904435ba0268f24b31dcf3"
format_stamp: "Formatted at 2026-09-26 20:20:10 on dist-test-slave-vnbh"
server_key: "7266b1aefe39b0a084bf0ad5ba4e4ab1"
server_key_iv: "88d76ec2fec29d2f40e08513b77750bb"
server_key_version: "encryptionkey@0"
I20260926 20:20:11.000806 20925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "0aab19aead904435ba0268f24b31dcf3"
format_stamp: "Formatted at 2026-09-26 20:20:10 on dist-test-slave-vnbh"
server_key: "7266b1aefe39b0a084bf0ad5ba4e4ab1"
server_key_iv: "88d76ec2fec29d2f40e08513b77750bb"
server_key_version: "encryptionkey@0"
I20260926 20:20:11.005080 20925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:20:11.008095 20940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:11.009768 20925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:11.009932 20925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "0aab19aead904435ba0268f24b31dcf3"
format_stamp: "Formatted at 2026-09-26 20:20:10 on dist-test-slave-vnbh"
server_key: "7266b1aefe39b0a084bf0ad5ba4e4ab1"
server_key_iv: "88d76ec2fec29d2f40e08513b77750bb"
server_key_version: "encryptionkey@0"
I20260926 20:20:11.010066 20925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:11.015129 20925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:11.018715 20925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:11.018949 20925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:11.026569 20925 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46397
I20260926 20:20:11.026602 20992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46397 every 8 connection(s)
I20260926 20:20:11.027570 20925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:11.030553 20993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:11.032430 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 20925
I20260926 20:20:11.032609 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:11.032959 32223 external_mini_cluster.cc:1428] Setting key 584c9b84d4139a8aae9520ff9064609b
I20260926 20:20:11.037448 20993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3: Bootstrap starting.
I20260926 20:20:11.038424 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:11.034420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:56833 (local address 127.0.0.1:46397)
0926 20:20:11.034918 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:11.034940 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:11.034978 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:20:11.035674 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:11.035692 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:11.035983 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:11.036337 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:11.037172 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:11.037177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:11.037188 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:11.037199 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:11.037251 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:11.037270 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:11.037454 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:11.037524 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:20:11.037691 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":160,"threads_started":1}
I20260926 20:20:11.039928 20993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:11.040788 20993 log.cc:824] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:11.041417 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:11.042972 20993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3: No bootstrap required, opened a new log
I20260926 20:20:11.045401 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:11.043563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:38979 (local address 127.0.0.1:46397)
0926 20:20:11.043678 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:11.043683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:11.043866 (+ 183us) server_negotiation.cc:407] Connection header received
0926 20:20:11.044004 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:11.044007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:11.044039 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:11.044113 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:11.044887 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:11.044890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:11.044891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:11.044902 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:11.044928 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:11.044938 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:11.045017 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:11.045161 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 20:20:11.045242 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:20:11.047354 20993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab19aead904435ba0268f24b31dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } }
I20260926 20:20:11.047605 20993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:11.047664 20993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab19aead904435ba0268f24b31dcf3, State: Initialized, Role: FOLLOWER
I20260926 20:20:11.048118 20993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab19aead904435ba0268f24b31dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } }
I20260926 20:20:11.048274 20993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:11.048344 20993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:11.048444 20993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:11.049453 20993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab19aead904435ba0268f24b31dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } }
I20260926 20:20:11.049796 20993 leader_election.cc:302] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aab19aead904435ba0268f24b31dcf3; no voters:
I20260926 20:20:11.050137 20993 leader_election.cc:288] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:11.050211 21003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:11.050421 21003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [term 1 LEADER]: Becoming Leader. State: Replica: 0aab19aead904435ba0268f24b31dcf3, State: Running, Role: LEADER
I20260926 20:20:11.050776 21003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab19aead904435ba0268f24b31dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } }
I20260926 20:20:11.051491 20993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:11.052525 21005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aab19aead904435ba0268f24b31dcf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab19aead904435ba0268f24b31dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } } }
I20260926 20:20:11.052650 21005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:11.052991 21004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aab19aead904435ba0268f24b31dcf3. Latest consensus state: current_term: 1 leader_uuid: "0aab19aead904435ba0268f24b31dcf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab19aead904435ba0268f24b31dcf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46397 } } }
I20260926 20:20:11.053049 21012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:11.053812 21004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:11.055948 21012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:11.062340 21012 catalog_manager.cc:1347] Generated new cluster ID: 5a9cf83f0fa74f64bacb4543070e1a18
I20260926 20:20:11.062433 21012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:11.072122 21012 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:11.072523 21012 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:11.078446 21012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aab19aead904435ba0268f24b31dcf3: Generated new TSK 0
I20260926 20:20:11.079308 21012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:11.082227 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 20925
2026-09-26T20:20:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:11 dist-test-slave-vnbh krb5kdc[21024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:11 dist-test-slave-vnbh krb5kdc[21024](info): set up 2 sockets
Sep 26 20:20:11 dist-test-slave-vnbh krb5kdc[21024](info): commencing operation
krb5kdc: starting...
W20260926 20:20:12.170657 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:12 dist-test-slave-vnbh krb5kdc[21024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:12.309121 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:12.441493 21037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:12.441829 21037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:12.441918 21037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:12.446065 21037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:12.446156 21037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:12.446197 21037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:12.446256 21037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:12.446302 21037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:12.451710 21037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:12.452868 21037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:12.454005 21037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:12.463227 21045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:12.463241 21043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:12.463414 21037 server_base.cc:1034] running on GCE node
W20260926 20:20:12.463227 21042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:12.464272 21037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:12.465696 21037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:12.466936 21037 hybrid_clock.cc:648] HybridClock initialized: now 1790454012466906 us; error 48 us; skew 500 ppm
Sep 26 20:20:12 dist-test-slave-vnbh krb5kdc[21024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:12.470259 21037 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:12.471252 21037 webserver.cc:460] Webserver started at http://127.0.0.1:35881/ using document root <none> and password file <none>
I20260926 20:20:12.471887 21037 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:12.471987 21037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:12.472244 21037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:12.474565 21037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "e1299492f6be44339eaaf2d50dcd0b79"
format_stamp: "Formatted at 2026-09-26 20:20:12 on dist-test-slave-vnbh"
server_key: "c71e087dba3624f5e4bdcbd1287d9f9c"
server_key_iv: "8b13494b85cc5c29d3b3b83afccf025b"
server_key_version: "encryptionkey@0"
I20260926 20:20:12.475211 21037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "e1299492f6be44339eaaf2d50dcd0b79"
format_stamp: "Formatted at 2026-09-26 20:20:12 on dist-test-slave-vnbh"
server_key: "c71e087dba3624f5e4bdcbd1287d9f9c"
server_key_iv: "8b13494b85cc5c29d3b3b83afccf025b"
server_key_version: "encryptionkey@0"
I20260926 20:20:12.479576 21037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:12.482519 21052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:12.484066 21037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:12.484241 21037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "e1299492f6be44339eaaf2d50dcd0b79"
format_stamp: "Formatted at 2026-09-26 20:20:12 on dist-test-slave-vnbh"
server_key: "c71e087dba3624f5e4bdcbd1287d9f9c"
server_key_iv: "8b13494b85cc5c29d3b3b83afccf025b"
server_key_version: "encryptionkey@0"
I20260926 20:20:12.484378 21037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:12.503942 21037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:12.507798 21037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:12.508033 21037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:12.516263 21037 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41259
I20260926 20:20:12.516263 21104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41259 every 8 connection(s)
I20260926 20:20:12.517464 21037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:12.520852 21105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:12.526684 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 21037
I20260926 20:20:12.526942 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:12.527338 32223 external_mini_cluster.cc:1428] Setting key ed342257901c0edfce97e1fb0257b5b6
I20260926 20:20:12.527761 21105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79: Bootstrap starting.
I20260926 20:20:12.530690 21105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:12.531634 21105 log.cc:824] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:12.534189 21105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79: No bootstrap required, opened a new log
Sep 26 20:20:12 dist-test-slave-vnbh krb5kdc[21024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:20:12.539441 21105 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1299492f6be44339eaaf2d50dcd0b79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260926 20:20:12.539830 21105 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:12.539928 21105 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1299492f6be44339eaaf2d50dcd0b79, State: Initialized, Role: FOLLOWER
I20260926 20:20:12.540553 21105 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1299492f6be44339eaaf2d50dcd0b79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260926 20:20:12.540746 21105 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:12.540863 21105 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:12.540977 21105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:12.542203 21105 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1299492f6be44339eaaf2d50dcd0b79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260926 20:20:12.542644 21105 leader_election.cc:302] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1299492f6be44339eaaf2d50dcd0b79; no voters:
I20260926 20:20:12.543051 21105 leader_election.cc:288] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:12.543169 21110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:12.543408 21110 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [term 1 LEADER]: Becoming Leader. State: Replica: e1299492f6be44339eaaf2d50dcd0b79, State: Running, Role: LEADER
I20260926 20:20:12.543872 21110 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1299492f6be44339eaaf2d50dcd0b79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260926 20:20:12.544512 21105 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:12.545928 21111 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1299492f6be44339eaaf2d50dcd0b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1299492f6be44339eaaf2d50dcd0b79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } } }
I20260926 20:20:12.546022 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:12.528968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:46595 (local address 127.0.0.1:41259)
0926 20:20:12.529503 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:12.529526 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:12.529572 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:20:12.530482 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:12.530509 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:12.531042 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:12.531448 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:12.532416 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:12.533536 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:12.534875 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:12.535416 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:12.535540 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:12.538985 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:12.538991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:12.539008 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:12.539041 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:12.541532 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:12.542144 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:12.542150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:12.542158 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:12.542308 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:12.542741 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:12.542745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:12.542746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:12.543429 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:12.544340 (+ 911us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:12.544533 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 20:20:12.544816 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":136,"threads_started":1}
I20260926 20:20:12.546072 21111 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:12.546340 21111 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1299492f6be44339eaaf2d50dcd0b79. Latest consensus state: current_term: 1 leader_uuid: "e1299492f6be44339eaaf2d50dcd0b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1299492f6be44339eaaf2d50dcd0b79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } } }
I20260926 20:20:12.546430 21111 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:12.546818 21120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:12.549576 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:12.550685 21120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:12.557772 21120 catalog_manager.cc:1347] Generated new cluster ID: 75a11df08c2d41a7aefec254b17feb3c
I20260926 20:20:12.557854 21120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:12.562348 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:12.552038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:54145 (local address 127.0.0.1:41259)
0926 20:20:12.552158 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:12.552163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:12.553047 (+ 884us) server_negotiation.cc:407] Connection header received
0926 20:20:12.553209 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:12.553212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:12.553332 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:12.553413 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:12.554128 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:12.554732 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:12.555836 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:12.556146 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:12.556281 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:12.558222 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:12.558226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:12.558229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:12.558258 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:12.560177 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:12.560767 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:12.560770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:12.560772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:12.560857 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:12.561248 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:12.561252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:12.561254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:12.561500 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:12.561629 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:12.562059 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 20:20:12.562172 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:20:12.594606 21120 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:12.596035 21120 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:12.604391 21120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1299492f6be44339eaaf2d50dcd0b79: Generated new TSK 0
I20260926 20:20:12.605427 21120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:12.614269 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 21037
W20260926 20:20:12.620330 21022 connection.cc:535] client connection to 127.0.0.1:41259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41259): Connection reset by peer (error 104)
2026-09-26T20:20:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1533 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:12 dist-test-slave-vnbh krb5kdc[21136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:12 dist-test-slave-vnbh krb5kdc[21136](info): set up 2 sockets
Sep 26 20:20:12 dist-test-slave-vnbh krb5kdc[21136](info): commencing operation
krb5kdc: starting...
W20260926 20:20:13.690688 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:13 dist-test-slave-vnbh krb5kdc[21136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:13.824504 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:13.938182 21149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:13.938482 21149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:13.938527 21149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:13.942679 21149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:13.942760 21149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:13.942780 21149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:13.942797 21149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:13.942813 21149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:13.948414 21149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:13.949658 21149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:13.950696 21149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:13.959304 21155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:13.959491 21149 server_base.cc:1034] running on GCE node
W20260926 20:20:13.959307 21154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:13.959332 21157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:13.960314 21149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:13.961649 21149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:13.962893 21149 hybrid_clock.cc:648] HybridClock initialized: now 1790454013962862 us; error 51 us; skew 500 ppm
Sep 26 20:20:13 dist-test-slave-vnbh krb5kdc[21136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:13.966030 21149 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:13.967070 21149 webserver.cc:460] Webserver started at http://127.0.0.1:34063/ using document root <none> and password file <none>
I20260926 20:20:13.967664 21149 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:13.967761 21149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:13.968007 21149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:13.970224 21149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "59875e674b0647f7bba3e05ab12b12c8"
format_stamp: "Formatted at 2026-09-26 20:20:13 on dist-test-slave-vnbh"
server_key: "0befb5867a1d7f4bf75d93cf91319bce"
server_key_iv: "f968191ee6e2bd8092bdeff8067db386"
server_key_version: "encryptionkey@0"
I20260926 20:20:13.970816 21149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "59875e674b0647f7bba3e05ab12b12c8"
format_stamp: "Formatted at 2026-09-26 20:20:13 on dist-test-slave-vnbh"
server_key: "0befb5867a1d7f4bf75d93cf91319bce"
server_key_iv: "f968191ee6e2bd8092bdeff8067db386"
server_key_version: "encryptionkey@0"
I20260926 20:20:13.974924 21149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:20:13.977897 21164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:13.979312 21149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:13.979473 21149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "59875e674b0647f7bba3e05ab12b12c8"
format_stamp: "Formatted at 2026-09-26 20:20:13 on dist-test-slave-vnbh"
server_key: "0befb5867a1d7f4bf75d93cf91319bce"
server_key_iv: "f968191ee6e2bd8092bdeff8067db386"
server_key_version: "encryptionkey@0"
I20260926 20:20:13.979604 21149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:13.995792 21149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:13.999526 21149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:13.999755 21149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:14.007875 21149 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39343
I20260926 20:20:14.007922 21216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39343 every 8 connection(s)
I20260926 20:20:14.008944 21149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:14.011420 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 21149
I20260926 20:20:14.011572 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:14.012065 32223 external_mini_cluster.cc:1428] Setting key 21c59fac50375561dd77b9e5bb1bb1e4
I20260926 20:20:14.012596 21217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:14.020920 21217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8: Bootstrap starting.
I20260926 20:20:14.023312 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:14.013900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:53875 (local address 127.0.0.1:39343)
0926 20:20:14.014384 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:14.014403 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:14.014451 (+ 48us) server_negotiation.cc:407] Connection header received
0926 20:20:14.015400 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:14.015423 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:14.015863 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:14.016195 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:14.016740 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:14.017979 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:14.019448 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:14.019965 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:14.020093 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:14.021226 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:14.021231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:14.021246 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:14.021741 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:14.021808 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:14.021882 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:14.021998 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:14.022232 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 20:20:14.022409 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":170,"threads_started":1}
I20260926 20:20:14.023828 21217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:14.024806 21217 log.cc:824] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:14.027357 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:14.028045 21217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8: No bootstrap required, opened a new log
I20260926 20:20:14.032845 21217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59875e674b0647f7bba3e05ab12b12c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20260926 20:20:14.033219 21217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:14.033317 21217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59875e674b0647f7bba3e05ab12b12c8, State: Initialized, Role: FOLLOWER
I20260926 20:20:14.033942 21217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59875e674b0647f7bba3e05ab12b12c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20260926 20:20:14.034150 21217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:14.034297 21217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:14.034462 21217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:14.035130 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:14.029946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:45153 (local address 127.0.0.1:39343)
0926 20:20:14.030097 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:14.030102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:14.030271 (+ 169us) server_negotiation.cc:407] Connection header received
0926 20:20:14.030375 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:14.030379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:14.030437 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:14.030540 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:14.031067 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:14.031993 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:14.032880 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:14.033266 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:14.033414 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:14.034527 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:14.034532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:14.034535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:14.034554 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:14.034591 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:14.034607 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:14.034721 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:14.034926 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 20:20:14.034970 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:20:14.035971 21217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59875e674b0647f7bba3e05ab12b12c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20260926 20:20:14.036406 21217 leader_election.cc:302] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59875e674b0647f7bba3e05ab12b12c8; no voters:
I20260926 20:20:14.036821 21217 leader_election.cc:288] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:14.036949 21227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:14.037206 21227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [term 1 LEADER]: Becoming Leader. State: Replica: 59875e674b0647f7bba3e05ab12b12c8, State: Running, Role: LEADER
I20260926 20:20:14.037614 21227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59875e674b0647f7bba3e05ab12b12c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20260926 20:20:14.038336 21217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:14.039353 21229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59875e674b0647f7bba3e05ab12b12c8. Latest consensus state: current_term: 1 leader_uuid: "59875e674b0647f7bba3e05ab12b12c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59875e674b0647f7bba3e05ab12b12c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } } }
I20260926 20:20:14.039481 21229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:14.039934 21228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59875e674b0647f7bba3e05ab12b12c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59875e674b0647f7bba3e05ab12b12c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } } }
I20260926 20:20:14.040033 21228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:14.040288 21236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:14.043648 21236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:14.049229 21236 catalog_manager.cc:1347] Generated new cluster ID: 6f3d70f9cdec43bb8345b11034b5f25b
I20260926 20:20:14.049352 21236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:14.058020 21236 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:14.059000 21236 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:14.068238 21236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59875e674b0647f7bba3e05ab12b12c8: Generated new TSK 0
I20260926 20:20:14.069025 21236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:14.074559 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 21149
W20260926 20:20:14.080293 21134 connection.cc:535] client connection to 127.0.0.1:39343 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39343): Connection reset by peer (error 104)
2026-09-26T20:20:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:14 dist-test-slave-vnbh krb5kdc[21248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:14 dist-test-slave-vnbh krb5kdc[21248](info): set up 2 sockets
Sep 26 20:20:14 dist-test-slave-vnbh krb5kdc[21248](info): commencing operation
krb5kdc: starting...
W20260926 20:20:15.118841 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:15 dist-test-slave-vnbh krb5kdc[21248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:20:15.250905 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:15.363611 21261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:15.363970 21261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:15.364050 21261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:15.368224 21261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:15.368338 21261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:15.368378 21261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:15.368415 21261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:15.368454 21261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:15.374140 21261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:15.375320 21261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:15.376467 21261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:15.385706 21266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:15.385660 21269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:15.385720 21261 server_base.cc:1034] running on GCE node
W20260926 20:20:15.385840 21267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:15.386516 21261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:15.387760 21261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:15.388931 21261 hybrid_clock.cc:648] HybridClock initialized: now 1790454015388916 us; error 26 us; skew 500 ppm
Sep 26 20:20:15 dist-test-slave-vnbh krb5kdc[21248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:15.391860 21261 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:20:15.392839 21261 webserver.cc:460] Webserver started at http://127.0.0.1:45361/ using document root <none> and password file <none>
I20260926 20:20:15.393466 21261 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:15.393559 21261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:15.393800 21261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:15.395852 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "58a48829345c4c8abcdd3137031b67b0"
format_stamp: "Formatted at 2026-09-26 20:20:15 on dist-test-slave-vnbh"
server_key: "0b8db35d4211b79a9a6e764456fae0d0"
server_key_iv: "453202a067cc64a656f75c0a8239199c"
server_key_version: "encryptionkey@0"
I20260926 20:20:15.396415 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "58a48829345c4c8abcdd3137031b67b0"
format_stamp: "Formatted at 2026-09-26 20:20:15 on dist-test-slave-vnbh"
server_key: "0b8db35d4211b79a9a6e764456fae0d0"
server_key_iv: "453202a067cc64a656f75c0a8239199c"
server_key_version: "encryptionkey@0"
I20260926 20:20:15.400238 21261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 20:20:15.402674 21276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:15.403957 21261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:20:15.404100 21261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "58a48829345c4c8abcdd3137031b67b0"
format_stamp: "Formatted at 2026-09-26 20:20:15 on dist-test-slave-vnbh"
server_key: "0b8db35d4211b79a9a6e764456fae0d0"
server_key_iv: "453202a067cc64a656f75c0a8239199c"
server_key_version: "encryptionkey@0"
I20260926 20:20:15.404214 21261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:15.408890 21261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:15.412004 21261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:15.412218 21261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:15.419623 21261 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41627
I20260926 20:20:15.419668 21328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41627 every 8 connection(s)
I20260926 20:20:15.420776 21261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:15.424005 21329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:15.427496 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 21261
I20260926 20:20:15.427636 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:15.427935 32223 external_mini_cluster.cc:1428] Setting key 21a79977683b9db0b0445c6e7cd0cafa
I20260926 20:20:15.430315 21329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0: Bootstrap starting.
I20260926 20:20:15.432946 21329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:15.433575 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:15.429428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:50485 (local address 127.0.0.1:41627)
0926 20:20:15.429873 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:15.429888 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:15.429923 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:20:15.430610 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:15.430629 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:15.430934 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:15.431222 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:15.432088 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:15.432093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:15.432105 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:15.432116 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:15.432170 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:15.432189 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:15.432309 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:15.432432 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 20:20:15.432620 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":145,"threads_started":1}
I20260926 20:20:15.433887 21329 log.cc:824] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:15.435889 21329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0: No bootstrap required, opened a new log
I20260926 20:20:15.436372 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:15.440292 21329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a48829345c4c8abcdd3137031b67b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41627 } }
I20260926 20:20:15.440569 21329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:15.440573 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:15.438756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:38965 (local address 127.0.0.1:41627)
0926 20:20:15.438867 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:15.438870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:15.439057 (+ 187us) server_negotiation.cc:407] Connection header received
0926 20:20:15.439249 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:15.439252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:15.439284 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:15.439373 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:15.440158 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:15.440161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:15.440164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:15.440175 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:15.440210 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:15.440228 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:15.440317 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:15.440402 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:20:15.440445 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:20:15.440667 21329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a48829345c4c8abcdd3137031b67b0, State: Initialized, Role: FOLLOWER
I20260926 20:20:15.441160 21329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a48829345c4c8abcdd3137031b67b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41627 } }
I20260926 20:20:15.441336 21329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:15.441411 21329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:15.441541 21329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:15.442497 21329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a48829345c4c8abcdd3137031b67b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41627 } }
I20260926 20:20:15.442864 21329 leader_election.cc:302] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a48829345c4c8abcdd3137031b67b0; no voters:
I20260926 20:20:15.443172 21329 leader_election.cc:288] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:15.443322 21339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:15.443660 21339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [term 1 LEADER]: Becoming Leader. State: Replica: 58a48829345c4c8abcdd3137031b67b0, State: Running, Role: LEADER
I20260926 20:20:15.444067 21339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a48829345c4c8abcdd3137031b67b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41627 } }
I20260926 20:20:15.444480 21329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:15.446381 21340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a48829345c4c8abcdd3137031b67b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a48829345c4c8abcdd3137031b67b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41627 } } }
I20260926 20:20:15.446426 21341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a48829345c4c8abcdd3137031b67b0. Latest consensus state: current_term: 1 leader_uuid: "58a48829345c4c8abcdd3137031b67b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a48829345c4c8abcdd3137031b67b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41627 } } }
I20260926 20:20:15.446520 21340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:15.446529 21341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0 [sys.catalog]: This master's current role is: LEADER
W20260926 20:20:15.449342 21353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:15.449427 21353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:15.449506 21354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:15.450304 21354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:15.456735 21354 catalog_manager.cc:1347] Generated new cluster ID: 88025560e6174c359c2eb642239ea276
I20260926 20:20:15.456821 21354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:15.472246 21354 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:15.472638 21354 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:15.489358 21354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58a48829345c4c8abcdd3137031b67b0: Generated new TSK 0
I20260926 20:20:15.490320 21354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:15.498674 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 21261
2026-09-26T20:20:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:15 dist-test-slave-vnbh krb5kdc[21360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:15 dist-test-slave-vnbh krb5kdc[21360](info): set up 2 sockets
Sep 26 20:20:15 dist-test-slave-vnbh krb5kdc[21360](info): commencing operation
krb5kdc: starting...
W20260926 20:20:16.538710 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:16 dist-test-slave-vnbh krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:20:16.669963 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:40035
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:40035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:16.787410 21373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:16.787734 21373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:16.787813 21373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:16.791960 21373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:16.792061 21373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:16.792101 21373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:16.792160 21373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:16.792198 21373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:16.797847 21373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:40035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:16.799000 21373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:16.800109 21373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:16.809386 21378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:16.809422 21373 server_base.cc:1034] running on GCE node
W20260926 20:20:16.809664 21379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:16.809494 21381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:16.810442 21373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:16.811584 21373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:16.812758 21373 hybrid_clock.cc:648] HybridClock initialized: now 1790454016812735 us; error 37 us; skew 500 ppm
Sep 26 20:20:16 dist-test-slave-vnbh krb5kdc[21360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454016, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:16.823140 21373 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:20:16.824214 21373 webserver.cc:460] Webserver started at http://127.31.119.254:42825/ using document root <none> and password file <none>
I20260926 20:20:16.824860 21373 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:16.824928 21373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:16.825155 21373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:16.827364 21373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "cea83e01ee1547268589f8b0b47bb786"
format_stamp: "Formatted at 2026-09-26 20:20:16 on dist-test-slave-vnbh"
server_key: "bf71ee3ee0e6fa7302e6e9898fb69655"
server_key_iv: "0e977a13622a3f476dd19c416526e4ac"
server_key_version: "encryptionkey@0"
I20260926 20:20:16.827976 21373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "cea83e01ee1547268589f8b0b47bb786"
format_stamp: "Formatted at 2026-09-26 20:20:16 on dist-test-slave-vnbh"
server_key: "bf71ee3ee0e6fa7302e6e9898fb69655"
server_key_iv: "0e977a13622a3f476dd19c416526e4ac"
server_key_version: "encryptionkey@0"
I20260926 20:20:16.831972 21373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:16.834613 21388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:16.835893 21373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:20:16.836025 21373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "cea83e01ee1547268589f8b0b47bb786"
format_stamp: "Formatted at 2026-09-26 20:20:16 on dist-test-slave-vnbh"
server_key: "bf71ee3ee0e6fa7302e6e9898fb69655"
server_key_iv: "0e977a13622a3f476dd19c416526e4ac"
server_key_version: "encryptionkey@0"
I20260926 20:20:16.836140 21373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:16.852738 21373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:16.856709 21373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:16.856963 21373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:16.865005 21373 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:40035
I20260926 20:20:16.865016 21440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:40035 every 8 connection(s)
I20260926 20:20:16.866091 21373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:16.867487 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 21373
I20260926 20:20:16.867640 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:16.867997 32223 external_mini_cluster.cc:1428] Setting key 955bc414caccd05928ccc3a3a59cbc7f
I20260926 20:20:16.869683 21441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:16.876567 21441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786: Bootstrap starting.
Sep 26 20:20:16 dist-test-slave-vnbh krb5kdc[21360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:20:16.879467 21441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:16.880400 21441 log.cc:824] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:16.883170 21441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786: No bootstrap required, opened a new log
I20260926 20:20:16.884315 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:16.869533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59611 (local address 127.31.119.254:40035)
0926 20:20:16.869972 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:16.869997 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:16.870034 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:20:16.870721 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:16.870737 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:16.871080 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:16.871376 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:16.872282 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:16.873408 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:16.874653 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:16.875148 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:16.875266 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:16.878326 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:16.878332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:16.878348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:16.878381 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:16.880379 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:16.881154 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:16.881159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:16.881164 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:16.881536 (+ 372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:16.881949 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:16.881953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:16.881955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:16.882489 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:16.882694 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:16.883074 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 20:20:16.883327 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":158,"threads_started":1}
I20260926 20:20:16.887645 21441 raft_consensus.cc:348] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea83e01ee1547268589f8b0b47bb786" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40035 } }
I20260926 20:20:16.887987 21441 raft_consensus.cc:374] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:16.888036 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:16.888077 21441 raft_consensus.cc:729] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea83e01ee1547268589f8b0b47bb786, State: Initialized, Role: FOLLOWER
I20260926 20:20:16.888515 21441 consensus_queue.cc:260] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea83e01ee1547268589f8b0b47bb786" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40035 } }
I20260926 20:20:16.888638 21441 raft_consensus.cc:388] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:16.888679 21441 raft_consensus.cc:482] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:16.888785 21441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:16.889984 21441 raft_consensus.cc:504] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea83e01ee1547268589f8b0b47bb786" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40035 } }
I20260926 20:20:16.890414 21441 leader_election.cc:302] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cea83e01ee1547268589f8b0b47bb786; no voters:
I20260926 20:20:16.890877 21441 leader_election.cc:288] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:16.891016 21451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:16.891279 21451 raft_consensus.cc:686] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [term 1 LEADER]: Becoming Leader. State: Replica: cea83e01ee1547268589f8b0b47bb786, State: Running, Role: LEADER
I20260926 20:20:16.891738 21451 consensus_queue.cc:237] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea83e01ee1547268589f8b0b47bb786" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40035 } }
I20260926 20:20:16.892267 21441 sys_catalog.cc:564] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:16.893009 21453 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cea83e01ee1547268589f8b0b47bb786. Latest consensus state: current_term: 1 leader_uuid: "cea83e01ee1547268589f8b0b47bb786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea83e01ee1547268589f8b0b47bb786" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40035 } } }
I20260926 20:20:16.893147 21453 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:16.893532 21452 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cea83e01ee1547268589f8b0b47bb786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea83e01ee1547268589f8b0b47bb786" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 40035 } } }
I20260926 20:20:16.893658 21452 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786 [sys.catalog]: This master's current role is: LEADER
W20260926 20:20:16.898120 21466 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:16.898211 21466 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:16.898281 21460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:16.898957 21460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:16.900370 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:16.890452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57607 (local address 127.31.119.254:40035)
0926 20:20:16.890597 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:16.890602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:16.890770 (+ 168us) server_negotiation.cc:407] Connection header received
0926 20:20:16.890848 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:16.890851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:16.890903 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:16.890989 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:16.891982 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:16.892757 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:16.894102 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:16.894497 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:16.894620 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:16.896415 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:16.896419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:16.896424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:16.896453 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:16.898038 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:16.898597 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:16.898601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:16.898603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:16.898688 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:16.899035 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:16.899041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:16.899043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:16.899265 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:16.899407 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:16.900109 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 20:20:16.900210 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:20:16.904633 21460 catalog_manager.cc:1347] Generated new cluster ID: 343dc538c7f948668a376654c03eec2b
I20260926 20:20:16.904704 21460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:16.912952 21460 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:16.914263 21460 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:16.920910 21460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cea83e01ee1547268589f8b0b47bb786: Generated new TSK 0
I20260926 20:20:16.921787 21460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:16.930716 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 21373
W20260926 20:20:16.936200 21358 connection.cc:535] client connection to 127.31.119.254:40035 recv error: Network error: failed to read from TLS socket (remote: 127.31.119.254:40035): Connection reset by peer (error 104)
2026-09-26T20:20:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:16 dist-test-slave-vnbh krb5kdc[21472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:16 dist-test-slave-vnbh krb5kdc[21472](info): set up 2 sockets
Sep 26 20:20:16 dist-test-slave-vnbh krb5kdc[21472](info): commencing operation
krb5kdc: starting...
W20260926 20:20:17.966562 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:18 dist-test-slave-vnbh krb5kdc[21472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:20:18.097198 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:41283
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:41283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:18.211189 21485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:18.211474 21485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:18.211517 21485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:18.215699 21485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:18.215770 21485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:18.215790 21485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:18.215806 21485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:18.215821 21485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:18.221344 21485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:41283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:41283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:18.222519 21485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:18.223611 21485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:18.232715 21491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:18.232709 21490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:18.232879 21485 server_base.cc:1034] running on GCE node
W20260926 20:20:18.232715 21493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:18.233767 21485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:18.234957 21485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:18.236181 21485 hybrid_clock.cc:648] HybridClock initialized: now 1790454018236150 us; error 47 us; skew 500 ppm
Sep 26 20:20:18 dist-test-slave-vnbh krb5kdc[21472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454018, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:18.239181 21485 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:20:18.240187 21485 webserver.cc:460] Webserver started at http://127.31.119.254:44923/ using document root <none> and password file <none>
I20260926 20:20:18.240854 21485 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:18.240957 21485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:18.241338 21485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:18.243494 21485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "842c2d859bee478f85ac2cffb609340f"
format_stamp: "Formatted at 2026-09-26 20:20:18 on dist-test-slave-vnbh"
server_key: "37a1e4c4ee75f58e002d5e14d1661dd0"
server_key_iv: "54e1eade11ffc0636f07c7084e87069b"
server_key_version: "encryptionkey@0"
I20260926 20:20:18.244110 21485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "842c2d859bee478f85ac2cffb609340f"
format_stamp: "Formatted at 2026-09-26 20:20:18 on dist-test-slave-vnbh"
server_key: "37a1e4c4ee75f58e002d5e14d1661dd0"
server_key_iv: "54e1eade11ffc0636f07c7084e87069b"
server_key_version: "encryptionkey@0"
I20260926 20:20:18.248173 21485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:18.250810 21500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:18.252307 21485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:20:18.252419 21485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "842c2d859bee478f85ac2cffb609340f"
format_stamp: "Formatted at 2026-09-26 20:20:18 on dist-test-slave-vnbh"
server_key: "37a1e4c4ee75f58e002d5e14d1661dd0"
server_key_iv: "54e1eade11ffc0636f07c7084e87069b"
server_key_version: "encryptionkey@0"
I20260926 20:20:18.252523 21485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:18.269680 21485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:18.273262 21485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:18.273447 21485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:18.281009 21485 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:41283
I20260926 20:20:18.281036 21552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:41283 every 8 connection(s)
I20260926 20:20:18.282083 21485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:18.284276 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 21485
I20260926 20:20:18.284427 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:18.284724 32223 external_mini_cluster.cc:1428] Setting key 1d8bceeec45fdfa42a07743efb4c37fa
I20260926 20:20:18.285099 21553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:18.292604 21553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f: Bootstrap starting.
I20260926 20:20:18.295426 21553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:18.295754 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:18.286312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48819 (local address 127.31.119.254:41283)
0926 20:20:18.286762 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:18.286784 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:18.286829 (+ 45us) server_negotiation.cc:407] Connection header received
0926 20:20:18.287934 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:18.287960 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:18.288427 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:18.288818 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:18.289253 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:18.290764 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:18.291727 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:18.292431 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:18.292596 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:18.293598 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:18.293604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:18.293620 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:18.294151 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:18.294233 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:18.294263 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:18.294396 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:18.294629 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 20:20:18.294770 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":130,"threads_started":1}
I20260926 20:20:18.296278 21553 log.cc:824] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:18.298800 21553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f: No bootstrap required, opened a new log
I20260926 20:20:18.299448 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:18.303135 21553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c2d859bee478f85ac2cffb609340f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41283 } }
I20260926 20:20:18.303476 21553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:18.303552 21553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842c2d859bee478f85ac2cffb609340f, State: Initialized, Role: FOLLOWER
I20260926 20:20:18.304067 21553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c2d859bee478f85ac2cffb609340f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41283 } }
I20260926 20:20:18.304227 21553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:18.304317 21553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:18.304442 21553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:18.305799 21553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c2d859bee478f85ac2cffb609340f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41283 } }
I20260926 20:20:18.306211 21553 leader_election.cc:302] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842c2d859bee478f85ac2cffb609340f; no voters:
I20260926 20:20:18.306362 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:18.301729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52059 (local address 127.31.119.254:41283)
0926 20:20:18.301877 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:18.301881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:18.302062 (+ 181us) server_negotiation.cc:407] Connection header received
0926 20:20:18.302182 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:18.302184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:18.302259 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:18.302330 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:18.302656 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:18.303174 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:18.304360 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:18.304703 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:18.304806 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:18.305788 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:18.305791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:18.305793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:18.305806 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:18.305832 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:18.305843 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:18.305948 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:18.306118 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 20:20:18.306158 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:20:18.306596 21553 leader_election.cc:288] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:18.306696 21563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:18.306929 21563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [term 1 LEADER]: Becoming Leader. State: Replica: 842c2d859bee478f85ac2cffb609340f, State: Running, Role: LEADER
I20260926 20:20:18.307309 21563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c2d859bee478f85ac2cffb609340f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41283 } }
I20260926 20:20:18.307981 21553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:18.308998 21564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "842c2d859bee478f85ac2cffb609340f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c2d859bee478f85ac2cffb609340f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41283 } } }
I20260926 20:20:18.309128 21564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:18.309454 21565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842c2d859bee478f85ac2cffb609340f. Latest consensus state: current_term: 1 leader_uuid: "842c2d859bee478f85ac2cffb609340f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842c2d859bee478f85ac2cffb609340f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 41283 } } }
I20260926 20:20:18.309525 21572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:18.309540 21565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:18.312870 21572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:18.318482 21572 catalog_manager.cc:1347] Generated new cluster ID: 658ec7fb441a4e6e8caf92fde81628fa
I20260926 20:20:18.318569 21572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:18.331981 21572 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:18.333401 21572 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:18.348310 21572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842c2d859bee478f85ac2cffb609340f: Generated new TSK 0
I20260926 20:20:18.349062 21572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:18.355692 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 21485
W20260926 20:20:18.361374 21470 connection.cc:535] client connection to 127.31.119.254:41283 recv error: Network error: failed to read from TLS socket (remote: 127.31.119.254:41283): Connection reset by peer (error 104)
2026-09-26T20:20:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:18 dist-test-slave-vnbh krb5kdc[21584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:18 dist-test-slave-vnbh krb5kdc[21584](info): set up 2 sockets
Sep 26 20:20:18 dist-test-slave-vnbh krb5kdc[21584](info): commencing operation
krb5kdc: starting...
W20260926 20:20:19.470438 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:19 dist-test-slave-vnbh krb5kdc[21584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:20:19.600883 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:34049
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:19.715349 21597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:19.715741 21597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:19.715847 21597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:19.719976 21597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:19.720082 21597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:19.720122 21597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:19.720160 21597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:19.720191 21597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:19.725648 21597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:34049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:19.726874 21597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:19.727990 21597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:20:19.737399 21597 server_base.cc:1034] running on GCE node
W20260926 20:20:19.737360 21602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:19.737329 21603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:19.737357 21605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:19.738265 21597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:19.739558 21597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:19.740772 21597 hybrid_clock.cc:648] HybridClock initialized: now 1790454019740758 us; error 32 us; skew 500 ppm
Sep 26 20:20:19 dist-test-slave-vnbh krb5kdc[21584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454019, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:19.743885 21597 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:20:19.745007 21597 webserver.cc:460] Webserver started at http://127.31.119.254:46619/ using document root <none> and password file <none>
I20260926 20:20:19.745652 21597 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:19.745754 21597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:19.746006 21597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:19.748190 21597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "1a3bcb8b0a89418fb2704e8fc867094f"
format_stamp: "Formatted at 2026-09-26 20:20:19 on dist-test-slave-vnbh"
server_key: "4714f51ff55dd1e98e83c64218fae9ac"
server_key_iv: "a63c575d6104e1881cb36e1230e9a813"
server_key_version: "encryptionkey@0"
I20260926 20:20:19.748787 21597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "1a3bcb8b0a89418fb2704e8fc867094f"
format_stamp: "Formatted at 2026-09-26 20:20:19 on dist-test-slave-vnbh"
server_key: "4714f51ff55dd1e98e83c64218fae9ac"
server_key_iv: "a63c575d6104e1881cb36e1230e9a813"
server_key_version: "encryptionkey@0"
I20260926 20:20:19.752691 21597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:19.755403 21612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:19.756707 21597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:20:19.756853 21597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "1a3bcb8b0a89418fb2704e8fc867094f"
format_stamp: "Formatted at 2026-09-26 20:20:19 on dist-test-slave-vnbh"
server_key: "4714f51ff55dd1e98e83c64218fae9ac"
server_key_iv: "a63c575d6104e1881cb36e1230e9a813"
server_key_version: "encryptionkey@0"
I20260926 20:20:19.756978 21597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:19.762162 21597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:19.765928 21597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:19.766146 21597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:19.773595 21597 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:34049
I20260926 20:20:19.773684 21664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:34049 every 8 connection(s)
I20260926 20:20:19.774607 21597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:19.777942 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 21597
I20260926 20:20:19.778031 21665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:19.778087 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:19.778440 32223 external_mini_cluster.cc:1428] Setting key 6d3edf35df77fbc3a4a9ec6832d0c386
I20260926 20:20:19.785166 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:19.779961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54905 (local address 127.31.119.254:34049)
0926 20:20:19.780394 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:19.780414 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:19.780920 (+ 506us) server_negotiation.cc:407] Connection header received
0926 20:20:19.782084 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:19.782110 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:19.782508 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:19.782930 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:19.783656 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:19.783662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:19.783678 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:19.783690 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:19.783752 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:19.783778 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:19.783926 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:19.783999 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:20:19.784163 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":147,"threads_started":1}
I20260926 20:20:19.786293 21665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f: Bootstrap starting.
I20260926 20:20:19.788126 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:19.788977 21665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:19.790001 21665 log.cc:824] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:19.792353 21665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f: No bootstrap required, opened a new log
I20260926 20:20:19.792760 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:19.790614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51727 (local address 127.31.119.254:34049)
0926 20:20:19.790750 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:19.790755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:19.791004 (+ 249us) server_negotiation.cc:407] Connection header received
0926 20:20:19.791194 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:19.791197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:19.791242 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:19.791334 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:19.792157 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:19.792161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:19.792163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:19.792176 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:19.792205 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:19.792225 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:19.792321 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:19.792425 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 20:20:19.792465 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:20:19.796592 21665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3bcb8b0a89418fb2704e8fc867094f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34049 } }
I20260926 20:20:19.796900 21665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:19.796996 21665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a3bcb8b0a89418fb2704e8fc867094f, State: Initialized, Role: FOLLOWER
I20260926 20:20:19.797542 21665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3bcb8b0a89418fb2704e8fc867094f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34049 } }
I20260926 20:20:19.797714 21665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:19.797789 21665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:19.797907 21665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:19.799142 21665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3bcb8b0a89418fb2704e8fc867094f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34049 } }
I20260926 20:20:19.799525 21665 leader_election.cc:302] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a3bcb8b0a89418fb2704e8fc867094f; no voters:
I20260926 20:20:19.799899 21665 leader_election.cc:288] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:19.800038 21675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:19.800285 21675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [term 1 LEADER]: Becoming Leader. State: Replica: 1a3bcb8b0a89418fb2704e8fc867094f, State: Running, Role: LEADER
I20260926 20:20:19.800631 21675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3bcb8b0a89418fb2704e8fc867094f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34049 } }
I20260926 20:20:19.801236 21665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:19.802549 21677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a3bcb8b0a89418fb2704e8fc867094f. Latest consensus state: current_term: 1 leader_uuid: "1a3bcb8b0a89418fb2704e8fc867094f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3bcb8b0a89418fb2704e8fc867094f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34049 } } }
I20260926 20:20:19.802563 21676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a3bcb8b0a89418fb2704e8fc867094f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3bcb8b0a89418fb2704e8fc867094f" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 34049 } } }
I20260926 20:20:19.802711 21677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:19.802711 21676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:19.803469 21684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:19.806515 21684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:19.811841 21684 catalog_manager.cc:1347] Generated new cluster ID: 14ce8f46ddc14f44855a1142c5e4b4af
I20260926 20:20:19.811923 21684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:19.851789 21684 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:19.852128 21684 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:19.859591 21684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a3bcb8b0a89418fb2704e8fc867094f: Generated new TSK 0
I20260926 20:20:19.860607 21684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:19.875028 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 21597
2026-09-26T20:20:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1519 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:19 dist-test-slave-vnbh krb5kdc[21696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:19 dist-test-slave-vnbh krb5kdc[21696](info): set up 2 sockets
Sep 26 20:20:19 dist-test-slave-vnbh krb5kdc[21696](info): commencing operation
krb5kdc: starting...
W20260926 20:20:20.914582 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:21 dist-test-slave-vnbh krb5kdc[21696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:20:21.041266 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:45097
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:45097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:21.156162 21709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:21.156497 21709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:21.156546 21709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:21.160912 21709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:21.160991 21709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:21.161011 21709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:21.161029 21709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:21.161046 21709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:21.166927 21709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:45097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:45097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:21.168125 21709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:21.169262 21709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:21.178275 21714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:21.178267 21717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:21.178260 21715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:21.178647 21709 server_base.cc:1034] running on GCE node
I20260926 20:20:21.179190 21709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:21.180362 21709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:21.181547 21709 hybrid_clock.cc:648] HybridClock initialized: now 1790454021181520 us; error 42 us; skew 500 ppm
Sep 26 20:20:21 dist-test-slave-vnbh krb5kdc[21696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454021, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:21.184410 21709 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:20:21.185446 21709 webserver.cc:460] Webserver started at http://127.31.119.254:36195/ using document root <none> and password file <none>
I20260926 20:20:21.186035 21709 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:21.186129 21709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:21.186380 21709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:21.188700 21709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "cddace1ede39487b90e2ae3942a72a0a"
format_stamp: "Formatted at 2026-09-26 20:20:21 on dist-test-slave-vnbh"
server_key: "8f0eb71a2ca6f1b34e06a69bf701c0f2"
server_key_iv: "9ab26b9eb18a011b1b22994ff25b323a"
server_key_version: "encryptionkey@0"
I20260926 20:20:21.189409 21709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "cddace1ede39487b90e2ae3942a72a0a"
format_stamp: "Formatted at 2026-09-26 20:20:21 on dist-test-slave-vnbh"
server_key: "8f0eb71a2ca6f1b34e06a69bf701c0f2"
server_key_iv: "9ab26b9eb18a011b1b22994ff25b323a"
server_key_version: "encryptionkey@0"
I20260926 20:20:21.193485 21709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:20:21.196240 21724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:21.197655 21709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:21.197861 21709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "cddace1ede39487b90e2ae3942a72a0a"
format_stamp: "Formatted at 2026-09-26 20:20:21 on dist-test-slave-vnbh"
server_key: "8f0eb71a2ca6f1b34e06a69bf701c0f2"
server_key_iv: "9ab26b9eb18a011b1b22994ff25b323a"
server_key_version: "encryptionkey@0"
I20260926 20:20:21.198045 21709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:21.218483 21709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:21.221745 21709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:21.221975 21709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:21.229341 21709 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:45097
I20260926 20:20:21.229364 21776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:45097 every 8 connection(s)
I20260926 20:20:21.230345 21709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:21.233094 21777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:21.238267 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 21709
I20260926 20:20:21.238432 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:21.238767 32223 external_mini_cluster.cc:1428] Setting key a5249d30068cdb99642c8cb1dd2bead8
I20260926 20:20:21.239564 21777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a: Bootstrap starting.
I20260926 20:20:21.242184 21777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:21.242942 21777 log.cc:824] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:21.245004 21777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a: No bootstrap required, opened a new log
Sep 26 20:20:21 dist-test-slave-vnbh krb5kdc[21696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:20:21.249408 21777 raft_consensus.cc:348] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddace1ede39487b90e2ae3942a72a0a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45097 } }
I20260926 20:20:21.249730 21777 raft_consensus.cc:374] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:21.249804 21777 raft_consensus.cc:729] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddace1ede39487b90e2ae3942a72a0a, State: Initialized, Role: FOLLOWER
I20260926 20:20:21.250320 21777 consensus_queue.cc:260] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddace1ede39487b90e2ae3942a72a0a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45097 } }
I20260926 20:20:21.250504 21777 raft_consensus.cc:388] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:21.250602 21777 raft_consensus.cc:482] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:21.250722 21777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:21.251833 21777 raft_consensus.cc:504] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddace1ede39487b90e2ae3942a72a0a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45097 } }
I20260926 20:20:21.252199 21777 leader_election.cc:302] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cddace1ede39487b90e2ae3942a72a0a; no voters:
I20260926 20:20:21.252542 21777 leader_election.cc:288] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:21.252885 21782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:21.253149 21782 raft_consensus.cc:686] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [term 1 LEADER]: Becoming Leader. State: Replica: cddace1ede39487b90e2ae3942a72a0a, State: Running, Role: LEADER
I20260926 20:20:21.253598 21782 consensus_queue.cc:237] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddace1ede39487b90e2ae3942a72a0a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45097 } }
I20260926 20:20:21.254040 21777 sys_catalog.cc:564] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:21.255079 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:21.240291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44797 (local address 127.31.119.254:45097)
0926 20:20:21.240786 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:21.240801 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:21.240841 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:20:21.241639 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:21.241659 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:21.242024 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:21.242377 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:21.243243 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:21.244458 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:21.245486 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:21.245972 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:21.246157 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:21.249106 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:21.249111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:21.249125 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:21.249153 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:21.251284 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:21.251801 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:21.251805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:21.251810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:21.251966 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:21.252298 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:21.252302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:21.252303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:21.253024 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:21.253338 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:21.253524 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 20:20:21.254086 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":138,"threads_started":1}
I20260926 20:20:21.255367 21783 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cddace1ede39487b90e2ae3942a72a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddace1ede39487b90e2ae3942a72a0a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45097 } } }
I20260926 20:20:21.255481 21783 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:21.255700 21783 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cddace1ede39487b90e2ae3942a72a0a. Latest consensus state: current_term: 1 leader_uuid: "cddace1ede39487b90e2ae3942a72a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddace1ede39487b90e2ae3942a72a0a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45097 } } }
I20260926 20:20:21.255767 21783 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:21.256152 21792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:21.258698 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:21.259624 21792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:21.267268 21792 catalog_manager.cc:1347] Generated new cluster ID: a4621d59598747ada68e66ace05528f7
I20260926 20:20:21.267354 21792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:21.269425 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:21.260691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37969 (local address 127.31.119.254:45097)
0926 20:20:21.260812 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:21.260819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:21.261211 (+ 392us) server_negotiation.cc:407] Connection header received
0926 20:20:21.261376 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:21.261379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:21.261429 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:21.261524 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:21.262202 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:21.262726 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:21.263537 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:21.263864 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:21.263970 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:21.265783 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:21.265787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:21.265790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:21.265821 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:21.267717 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:21.268154 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:21.268158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:21.268160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:21.268251 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:21.268530 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:21.268533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:21.268534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:21.268720 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:21.268852 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:21.269085 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 20:20:21.269216 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:20:21.280283 21792 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:21.281255 21792 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:21.285503 21792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cddace1ede39487b90e2ae3942a72a0a: Generated new TSK 0
I20260926 20:20:21.286394 21792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:21.298117 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 21709
2026-09-26T20:20:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:21 dist-test-slave-vnbh krb5kdc[21808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:21 dist-test-slave-vnbh krb5kdc[21808](info): set up 2 sockets
Sep 26 20:20:21 dist-test-slave-vnbh krb5kdc[21808](info): commencing operation
krb5kdc: starting...
W20260926 20:20:22.334512 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:22 dist-test-slave-vnbh krb5kdc[21808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:20:22.476610 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:33465
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:33465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:22.592172 21821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:22.592456 21821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:22.592500 21821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:22.596596 21821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:22.596676 21821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:22.596696 21821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:22.596714 21821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:22.596729 21821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:22.602370 21821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:33465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:33465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:22.603524 21821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:22.604627 21821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:22.613636 21827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:22.613727 21821 server_base.cc:1034] running on GCE node
W20260926 20:20:22.613636 21826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:22.613636 21829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:22.614522 21821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:22.615839 21821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:22.617064 21821 hybrid_clock.cc:648] HybridClock initialized: now 1790454022617038 us; error 43 us; skew 500 ppm
Sep 26 20:20:22 dist-test-slave-vnbh krb5kdc[21808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454022, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:22.620322 21821 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:20:22.621400 21821 webserver.cc:460] Webserver started at http://127.31.119.254:44171/ using document root <none> and password file <none>
I20260926 20:20:22.621986 21821 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:22.622077 21821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:22.622294 21821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:22.624486 21821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "cbe478183f28445fb868cc69930a879a"
format_stamp: "Formatted at 2026-09-26 20:20:22 on dist-test-slave-vnbh"
server_key: "d035f599baf83fff432198cc1a48c37f"
server_key_iv: "8088dd7928c68ed14e4556ecd01653ef"
server_key_version: "encryptionkey@0"
I20260926 20:20:22.625130 21821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "cbe478183f28445fb868cc69930a879a"
format_stamp: "Formatted at 2026-09-26 20:20:22 on dist-test-slave-vnbh"
server_key: "d035f599baf83fff432198cc1a48c37f"
server_key_iv: "8088dd7928c68ed14e4556ecd01653ef"
server_key_version: "encryptionkey@0"
I20260926 20:20:22.629376 21821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:20:22.631976 21836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:22.633517 21821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:20:22.633661 21821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "cbe478183f28445fb868cc69930a879a"
format_stamp: "Formatted at 2026-09-26 20:20:22 on dist-test-slave-vnbh"
server_key: "d035f599baf83fff432198cc1a48c37f"
server_key_iv: "8088dd7928c68ed14e4556ecd01653ef"
server_key_version: "encryptionkey@0"
I20260926 20:20:22.633838 21821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:22.647099 21821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:22.650094 21821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:22.650295 21821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:22.657248 21821 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:33465
I20260926 20:20:22.657351 21888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:33465 every 8 connection(s)
I20260926 20:20:22.658253 21821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:22.661037 21889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:22.663558 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 21821
I20260926 20:20:22.663709 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:22.664023 32223 external_mini_cluster.cc:1428] Setting key fa1fdfb390d215d5690bb2e63062e955
I20260926 20:20:22.667474 21889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a: Bootstrap starting.
I20260926 20:20:22.670112 21889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:22.670951 21889 log.cc:824] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:22.673712 21889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a: No bootstrap required, opened a new log
I20260926 20:20:22.674541 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:22.665718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39597 (local address 127.31.119.254:33465)
0926 20:20:22.666207 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:22.666225 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:22.666264 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:20:22.666952 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:22.666971 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:22.667360 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:22.667749 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:22.668111 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:22.669192 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:22.670475 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:22.671099 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:22.671240 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:22.672247 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:22.672254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:22.672272 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:22.672814 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:22.672890 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:22.672924 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:22.673040 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:22.673217 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 20:20:22.673481 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":179,"threads_started":1}
I20260926 20:20:22.677835 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:22.678474 21889 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe478183f28445fb868cc69930a879a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33465 } }
I20260926 20:20:22.678726 21889 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:22.678802 21889 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe478183f28445fb868cc69930a879a, State: Initialized, Role: FOLLOWER
I20260926 20:20:22.679337 21889 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe478183f28445fb868cc69930a879a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33465 } }
I20260926 20:20:22.679473 21889 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:22.679562 21889 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:22.679690 21889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:22.681130 21889 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe478183f28445fb868cc69930a879a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33465 } }
I20260926 20:20:22.681553 21889 leader_election.cc:302] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbe478183f28445fb868cc69930a879a; no voters:
I20260926 20:20:22.681926 21889 leader_election.cc:288] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:22.682031 21899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:22.682250 21899 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [term 1 LEADER]: Becoming Leader. State: Replica: cbe478183f28445fb868cc69930a879a, State: Running, Role: LEADER
I20260926 20:20:22.682634 21899 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe478183f28445fb868cc69930a879a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33465 } }
I20260926 20:20:22.683422 21889 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:22.683948 21899 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbe478183f28445fb868cc69930a879a. Latest consensus state: current_term: 1 leader_uuid: "cbe478183f28445fb868cc69930a879a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe478183f28445fb868cc69930a879a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33465 } } }
I20260926 20:20:22.684147 21899 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:22.684435 21900 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbe478183f28445fb868cc69930a879a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe478183f28445fb868cc69930a879a" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 33465 } } }
I20260926 20:20:22.684621 21904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:22.685526 21900 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:22.685446 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:22.680005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53331 (local address 127.31.119.254:33465)
0926 20:20:22.680141 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:22.680145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:22.680293 (+ 148us) server_negotiation.cc:407] Connection header received
0926 20:20:22.680383 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:22.680386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:22.680442 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:22.680535 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:22.680810 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:22.681706 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:22.682906 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:22.683268 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:22.683406 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:22.684911 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:22.684918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:22.684924 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:22.684942 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:22.684975 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:22.684996 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:22.685046 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:22.685215 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:20:22.685271 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:20:22.688809 21904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:22.696029 21904 catalog_manager.cc:1347] Generated new cluster ID: f0bebeb86a2b4f85a06ad60cf1257d43
I20260926 20:20:22.696131 21904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:22.744958 21904 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:22.746018 21904 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:22.762085 21904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbe478183f28445fb868cc69930a879a: Generated new TSK 0
I20260926 20:20:22.762897 21904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:22.777261 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 21821
2026-09-26T20:20:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:22 dist-test-slave-vnbh krb5kdc[21920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:22 dist-test-slave-vnbh krb5kdc[21920](info): set up 2 sockets
Sep 26 20:20:22 dist-test-slave-vnbh krb5kdc[21920](info): commencing operation
krb5kdc: starting...
W20260926 20:20:23.818727 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:23 dist-test-slave-vnbh krb5kdc[21920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:20:23.945097 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:45875
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:45875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:24.059993 21933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:24.060308 21933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:24.060353 21933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:24.064520 21933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:24.064615 21933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:24.064635 21933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:24.064652 21933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:24.064668 21933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:24.070334 21933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:45875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:45875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:24.071465 21933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:24.072499 21933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:20:24.081532 21933 server_base.cc:1034] running on GCE node
W20260926 20:20:24.081481 21941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:24.081481 21938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:24.081475 21939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:24.082371 21933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:24.083578 21933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:24.084766 21933 hybrid_clock.cc:648] HybridClock initialized: now 1790454024084750 us; error 29 us; skew 500 ppm
Sep 26 20:20:24 dist-test-slave-vnbh krb5kdc[21920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454024, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:24.087803 21933 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:20:24.088722 21933 webserver.cc:460] Webserver started at http://127.31.119.254:41889/ using document root <none> and password file <none>
I20260926 20:20:24.089265 21933 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:24.089380 21933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:24.089560 21933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:24.091794 21933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "d608ae9a5ac8415daa4bc532b434fc71"
format_stamp: "Formatted at 2026-09-26 20:20:24 on dist-test-slave-vnbh"
server_key: "d461ab5ba8ef867d4ea3be368644d1c8"
server_key_iv: "50b2ff2ca76f5f2dd361d69f6f52ab75"
server_key_version: "encryptionkey@0"
I20260926 20:20:24.092391 21933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "d608ae9a5ac8415daa4bc532b434fc71"
format_stamp: "Formatted at 2026-09-26 20:20:24 on dist-test-slave-vnbh"
server_key: "d461ab5ba8ef867d4ea3be368644d1c8"
server_key_iv: "50b2ff2ca76f5f2dd361d69f6f52ab75"
server_key_version: "encryptionkey@0"
I20260926 20:20:24.096390 21933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:24.099114 21948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:24.100544 21933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:20:24.100664 21933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "d608ae9a5ac8415daa4bc532b434fc71"
format_stamp: "Formatted at 2026-09-26 20:20:24 on dist-test-slave-vnbh"
server_key: "d461ab5ba8ef867d4ea3be368644d1c8"
server_key_iv: "50b2ff2ca76f5f2dd361d69f6f52ab75"
server_key_version: "encryptionkey@0"
I20260926 20:20:24.100754 21933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:24.105542 21933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:24.108852 21933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:24.109005 21933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:24.116292 21933 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:45875
I20260926 20:20:24.116328 22000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:45875 every 8 connection(s)
I20260926 20:20:24.117357 21933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:24.120254 22001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:24.121716 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 21933
I20260926 20:20:24.121850 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:24.122141 32223 external_mini_cluster.cc:1428] Setting key fe4b817182c5ac576489941cac6efbe2
I20260926 20:20:24.126817 22001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71: Bootstrap starting.
I20260926 20:20:24.127544 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:24.123579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47215 (local address 127.31.119.254:45875)
0926 20:20:24.124067 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:24.124082 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:24.124122 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:20:24.124794 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:24.124812 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:24.125108 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:24.125497 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:24.126309 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:24.126315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:24.126334 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:24.126344 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:24.126401 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:24.126421 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:24.126570 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:24.126651 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:20:24.126814 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":144,"threads_started":1}
I20260926 20:20:24.129390 22001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:24.130223 22001 log.cc:824] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:24.130720 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:24.132495 22001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71: No bootstrap required, opened a new log
I20260926 20:20:24.134822 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:24.132936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54903 (local address 127.31.119.254:45875)
0926 20:20:24.133082 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:24.133087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:24.133319 (+ 232us) server_negotiation.cc:407] Connection header received
0926 20:20:24.133476 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:24.133479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:24.133518 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:24.133600 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:24.134357 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:24.134361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:24.134363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:24.134375 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:24.134405 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:24.134416 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:24.134506 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:24.134583 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 20:20:24.134672 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:20:24.137027 22001 raft_consensus.cc:348] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d608ae9a5ac8415daa4bc532b434fc71" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45875 } }
I20260926 20:20:24.137274 22001 raft_consensus.cc:374] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:24.137396 22001 raft_consensus.cc:729] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d608ae9a5ac8415daa4bc532b434fc71, State: Initialized, Role: FOLLOWER
I20260926 20:20:24.137869 22001 consensus_queue.cc:260] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d608ae9a5ac8415daa4bc532b434fc71" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45875 } }
I20260926 20:20:24.137997 22001 raft_consensus.cc:388] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:24.138039 22001 raft_consensus.cc:482] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:24.138105 22001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:24.139333 22001 raft_consensus.cc:504] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d608ae9a5ac8415daa4bc532b434fc71" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45875 } }
I20260926 20:20:24.139707 22001 leader_election.cc:302] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d608ae9a5ac8415daa4bc532b434fc71; no voters:
I20260926 20:20:24.140036 22001 leader_election.cc:288] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:24.140206 22011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:24.140437 22011 raft_consensus.cc:686] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [term 1 LEADER]: Becoming Leader. State: Replica: d608ae9a5ac8415daa4bc532b434fc71, State: Running, Role: LEADER
I20260926 20:20:24.140918 22011 consensus_queue.cc:237] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d608ae9a5ac8415daa4bc532b434fc71" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45875 } }
I20260926 20:20:24.141512 22001 sys_catalog.cc:564] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:24.142884 22012 sys_catalog.cc:455] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d608ae9a5ac8415daa4bc532b434fc71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d608ae9a5ac8415daa4bc532b434fc71" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45875 } } }
I20260926 20:20:24.142891 22013 sys_catalog.cc:455] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d608ae9a5ac8415daa4bc532b434fc71. Latest consensus state: current_term: 1 leader_uuid: "d608ae9a5ac8415daa4bc532b434fc71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d608ae9a5ac8415daa4bc532b434fc71" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 45875 } } }
I20260926 20:20:24.143029 22012 sys_catalog.cc:458] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:24.143029 22013 sys_catalog.cc:458] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:24.143404 22020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:24.146368 22020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:24.152259 22020 catalog_manager.cc:1347] Generated new cluster ID: 60749fc3205c46d0838a8a05f2ad9ece
I20260926 20:20:24.152338 22020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:24.176951 22020 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:24.177357 22020 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:24.195083 22020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d608ae9a5ac8415daa4bc532b434fc71: Generated new TSK 0
I20260926 20:20:24.195966 22020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:24.200428 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 21933
2026-09-26T20:20:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1423 ms)
[----------] 18 tests from AuthTokenIssuingTest (26253 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:24 dist-test-slave-vnbh krb5kdc[22032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:24 dist-test-slave-vnbh krb5kdc[22032](info): set up 2 sockets
Sep 26 20:20:24 dist-test-slave-vnbh krb5kdc[22032](info): commencing operation
krb5kdc: starting...
W20260926 20:20:25.250547 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:25 dist-test-slave-vnbh krb5kdc[22032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:20:25.381428 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:37243
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:37243,127.31.119.253:41281,127.31.119.252:38233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:25.496132 22045 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:20:25.496445 22045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:25.496470 22045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:25.496510 22045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:25.500659 22045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:25.500736 22045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:25.500757 22045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:25.500774 22045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:25.500790 22045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:25.506316 22045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:37243,127.31.119.253:41281,127.31.119.252:38233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:37243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:25.507404 22045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:25.508423 22045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:20:25.517238 22045 server_base.cc:1034] running on GCE node
W20260926 20:20:25.517191 22053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:25.517184 22051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:25.517181 22050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:25.518121 22045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:25.519330 22045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:25.520539 22045 hybrid_clock.cc:648] HybridClock initialized: now 1790454025520473 us; error 80 us; skew 500 ppm
Sep 26 20:20:25 dist-test-slave-vnbh krb5kdc[22032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:25.523386 22045 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:20:25.524221 22045 webserver.cc:460] Webserver started at http://127.31.119.254:46803/ using document root <none> and password file <none>
I20260926 20:20:25.524753 22045 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:25.524827 22045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:25.525007 22045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:25.527100 22045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3"
format_stamp: "Formatted at 2026-09-26 20:20:25 on dist-test-slave-vnbh"
server_key: "650d3abe3394f34b85c354883dd2af38"
server_key_iv: "6a369de7ff8175a1ee6e75d2b8e881ce"
server_key_version: "encryptionkey@0"
I20260926 20:20:25.527661 22045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3"
format_stamp: "Formatted at 2026-09-26 20:20:25 on dist-test-slave-vnbh"
server_key: "650d3abe3394f34b85c354883dd2af38"
server_key_iv: "6a369de7ff8175a1ee6e75d2b8e881ce"
server_key_version: "encryptionkey@0"
I20260926 20:20:25.531466 22045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:25.534219 22060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:25.535640 22045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:25.535779 22045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3"
format_stamp: "Formatted at 2026-09-26 20:20:25 on dist-test-slave-vnbh"
server_key: "650d3abe3394f34b85c354883dd2af38"
server_key_iv: "6a369de7ff8175a1ee6e75d2b8e881ce"
server_key_version: "encryptionkey@0"
I20260926 20:20:25.535900 22045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:25.551857 22045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:25.555450 22045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:25.555663 22045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:25.563381 22045 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:37243
I20260926 20:20:25.563704 22112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:37243 every 8 connection(s)
I20260926 20:20:25.564409 22045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:25.567428 22113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:25.567968 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 22045
I20260926 20:20:25.568158 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:25.568457 32223 external_mini_cluster.cc:1428] Setting key 4f27109419bed961afe97ea217f88512
I20260926 20:20:25.572181 22113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.119.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.253@KRBTEST.COM" created.
Sep 26 20:20:25 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:20:25.590478 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.574898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:37243 (local address 127.0.0.1:49788)
0926 20:20:25.575427 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.575481 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:25.576410 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:25.577246 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:25.577251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:25.577677 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:25.577953 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:25.577968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.579349 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.579352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:25.580313 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:25.580326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.581089 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.581096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:25.581244 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:25.582011 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:25.582029 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:25.583960 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:25.586885 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:25.586895 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:25.586905 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:25.587168 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:25.587698 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:25.587701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:25.587702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:25.587846 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:25.588764 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:25.588771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:25.588951 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 20:20:25.589123 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 20:20:25.589602 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":88,"thread_start_us":154,"threads_started":1}
I20260926 20:20:25.590477 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.574899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.31.119.254:37243)
0926 20:20:25.575481 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:25.575497 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:25.575618 (+ 121us) server_negotiation.cc:407] Connection header received
0926 20:20:25.576763 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:25.576769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:25.577037 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:25.577258 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:25.578132 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.579192 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.580450 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.580911 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.581054 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:25.584155 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:25.584167 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:25.584187 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:25.584224 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:25.586699 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:25.587318 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:25.587327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:25.587331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:25.587503 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:25.587959 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:25.587965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:25.587968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:25.588621 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:25.588763 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:25.589237 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 20:20:25.589498 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"spinlock_wait_cycles":14720,"thread_start_us":148,"threads_started":1}
I20260926 20:20:25.593005 22113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:25.593997 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.593675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:41281 (local address 127.0.0.1:43700)
0926 20:20:25.593818 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.593900 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 20:20:25.594216 22061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.119.253:41281: connect: Connection refused (error 111)
I20260926 20:20:25.595283 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.595069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:41281 (local address 127.0.0.1:43702)
0926 20:20:25.595171 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.595222 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 20:20:25.595504 22113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.253:41281: Network error: Client connection negotiation failed: client connection to 127.31.119.253:41281: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab.
Entry for principal kudu/127.31.119.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab.
I20260926 20:20:25.600512 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.253:41281
--webserver_interface=127.31.119.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab
--principal=kudu/127.31.119.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:37243,127.31.119.253:41281,127.31.119.252:38233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:20:25.644788 22113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } attempt: 1
I20260926 20:20:25.645797 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.645471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:41281 (local address 127.0.0.1:43708)
0926 20:20:25.645648 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.645717 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 20:20:25.646564 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.646354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:41281 (local address 127.0.0.1:43712)
0926 20:20:25.646455 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.646508 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 20:20:25.646692 22113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.253:41281: Network error: Client connection negotiation failed: client connection to 127.31.119.253:41281: connect: Connection refused (error 111)
I20260926 20:20:25.714881 22113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } attempt: 2
I20260926 20:20:25.715680 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.715391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:41281 (local address 127.0.0.1:43728)
0926 20:20:25.715538 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.715605 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 20:20:25.716423 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.716192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:41281 (local address 127.0.0.1:43738)
0926 20:20:25.716318 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.716367 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:41281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 20:20:25.716552 22113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.253:41281: Network error: Client connection negotiation failed: client connection to 127.31.119.253:41281: connect: Connection refused (error 111)
W20260926 20:20:25.719202 22120 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:20:25.719542 22120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:25.719601 22120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:25.719677 22120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:25.723991 22120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:25.724092 22120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:25.724131 22120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:25.724169 22120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:25.724202 22120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:25.729918 22120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:37243,127.31.119.253:41281,127.31.119.252:38233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab
--principal=kudu/127.31.119.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.253:41281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.119.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:25.731081 22120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:25.732136 22120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:25.741169 22125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:25.741251 22120 server_base.cc:1034] running on GCE node
W20260926 20:20:25.741169 22126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:25.741169 22128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:25.742029 22120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:25.743124 22120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:25.744299 22120 hybrid_clock.cc:648] HybridClock initialized: now 1790454025744279 us; error 32 us; skew 500 ppm
Sep 26 20:20:25 dist-test-slave-vnbh krb5kdc[22032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:25.747299 22120 init.cc:377] Logged in from keytab as kudu/127.31.119.253@KRBTEST.COM (short username kudu)
I20260926 20:20:25.748194 22120 webserver.cc:460] Webserver started at http://127.31.119.253:43913/ using document root <none> and password file <none>
I20260926 20:20:25.748806 22120 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:25.748869 22120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:25.749039 22120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:25.751165 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/data/instance:
uuid: "888b9f66d431411ca1a54cb94480d416"
format_stamp: "Formatted at 2026-09-26 20:20:25 on dist-test-slave-vnbh"
server_key: "41d1f4640263d1b6a48cc2226f3543f2"
server_key_iv: "a354bb73e35abb6a0c838fa0666b9405"
server_key_version: "encryptionkey@0"
I20260926 20:20:25.751703 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/wal/instance:
uuid: "888b9f66d431411ca1a54cb94480d416"
format_stamp: "Formatted at 2026-09-26 20:20:25 on dist-test-slave-vnbh"
server_key: "41d1f4640263d1b6a48cc2226f3543f2"
server_key_iv: "a354bb73e35abb6a0c838fa0666b9405"
server_key_version: "encryptionkey@0"
I20260926 20:20:25.755506 22120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:25.758116 22135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:25.759364 22120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:20:25.759503 22120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/wal
uuid: "888b9f66d431411ca1a54cb94480d416"
format_stamp: "Formatted at 2026-09-26 20:20:25 on dist-test-slave-vnbh"
server_key: "41d1f4640263d1b6a48cc2226f3543f2"
server_key_iv: "a354bb73e35abb6a0c838fa0666b9405"
server_key_version: "encryptionkey@0"
I20260926 20:20:25.759609 22120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:25.777419 22120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:25.780706 22120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:25.780949 22120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:25.788064 22120 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.253:41281
I20260926 20:20:25.788357 22187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.253:41281 every 8 connection(s)
I20260926 20:20:25.789057 22120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
I20260926 20:20:25.791893 22188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:25.795725 22188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:25.797768 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 22120
I20260926 20:20:25.797905 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-1/wal/instance
I20260926 20:20:25.798182 32223 external_mini_cluster.cc:1428] Setting key 6bfbde4e2849fb9c8ea6e808451f69d8
WARNING: no policy specified for kudu/127.31.119.252@KRBTEST.COM; defaulting to no policy
I20260926 20:20:25.807724 22113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.252@KRBTEST.COM" created.
Sep 26 20:20:25 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
Sep 26 20:20:25 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
I20260926 20:20:25.815039 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.797754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.31.119.254:37243)
0926 20:20:25.797913 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:25.797917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:25.798866 (+ 949us) server_negotiation.cc:407] Connection header received
0926 20:20:25.799807 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:25.799814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:25.799883 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:25.799996 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:25.801219 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.801901 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.804338 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.804768 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.804892 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:25.810144 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:25.810150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:25.810156 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:25.810190 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:25.812192 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:25.812947 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:25.812950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:25.812952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:25.813061 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:25.813522 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:25.813525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:25.813527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:25.813833 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:25.813971 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:25.814690 (+ 719us) server_negotiation.cc:299] Negotiation successful
0926 20:20:25.814822 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:20:25.816315 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.798172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:37243 (local address 127.0.0.1:49796)
0926 20:20:25.798695 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.798748 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:25.799593 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:25.800138 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:25.800144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:25.800478 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:25.801069 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:25.801081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.802702 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.802707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:25.804182 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:25.804208 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.804982 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.804989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:25.805191 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:25.805872 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:25.805889 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:25.809976 (+ 4087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:25.812361 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:25.812367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:25.812387 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:25.812806 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:25.813169 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:25.813172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:25.813174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:25.813401 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:25.813983 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:25.813993 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:25.814363 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 20:20:25.814609 (+ 246us) client_negotiation.cc:240] Negotiation successful
0926 20:20:25.815017 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":143,"threads_started":1}
I20260926 20:20:25.818130 22188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:25.819345 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.808223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:41281 (local address 127.0.0.1:43740)
0926 20:20:25.808337 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.808353 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:25.808441 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:25.809132 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:25.809134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:25.809367 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:25.809494 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:25.809502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.810672 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.810678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:25.811410 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:25.811425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.812312 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.812317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:25.812435 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:25.812917 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:25.812939 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:25.815495 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:25.817616 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:25.817622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:25.817624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:25.817826 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:25.818119 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:25.818122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:25.818124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:25.818209 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:25.818763 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:25.818769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:25.818881 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 20:20:25.819015 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 20:20:25.819167 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 20:20:25.820292 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.808366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.31.119.253:41281)
0926 20:20:25.808693 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:25.808702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:25.808724 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:20:25.808761 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:25.808765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:25.808977 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:25.809154 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:25.809616 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.810537 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.811561 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.812157 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.812299 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:25.815628 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:25.815637 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:25.815651 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:25.815679 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:25.817514 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:25.817932 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:25.817938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:25.817940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:25.818021 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:25.818326 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:25.818329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:25.818331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:25.818638 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:25.818779 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:25.819898 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0926 20:20:25.820055 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":110,"threads_started":1}
I20260926 20:20:25.822707 22113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:25.823504 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.823273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45204)
0926 20:20:25.823377 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.823430 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
Sep 26 20:20:25 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
I20260926 20:20:25.824384 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.824139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45220)
0926 20:20:25.824280 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.824330 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 20:20:25.824608 22113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:38233: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab.
Entry for principal kudu/127.31.119.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab.
I20260926 20:20:25.827857 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.818687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:41281 (local address 127.0.0.1:43752)
0926 20:20:25.818914 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.818928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:25.819007 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:25.819445 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:25.819447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:25.819575 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:25.819664 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:25.819669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.820746 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.820750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:25.821573 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:25.821585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.822086 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.822089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:25.822178 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:25.822685 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:25.822702 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:25.824242 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:25.826342 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:25.826345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:25.826347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:25.826568 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:25.826879 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:25.826882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:25.826883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:25.826961 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:25.827348 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:25.827351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:25.827433 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 20:20:25.827528 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:20:25.827706 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260926 20:20:25.828186 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.818897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.31.119.253:41281)
0926 20:20:25.819228 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:25.819237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:25.819251 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:20:25.819284 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:25.819287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:25.819326 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:25.819425 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:25.819834 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.820635 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.821714 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:25.821962 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:25.822091 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:25.824397 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:25.824400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:25.824403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:25.824430 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:25.826240 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:25.826707 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:25.826710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:25.826711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:25.826792 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:25.827041 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:25.827043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:25.827044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:25.827235 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:25.827314 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:25.827633 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 20:20:25.827794 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":129,"threads_started":1}
I20260926 20:20:25.828781 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.252:38233
--webserver_interface=127.31.119.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab
--principal=kudu/127.31.119.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:40169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:37243,127.31.119.253:41281,127.31.119.252:38233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:20:25.828967 22188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:25.829795 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.829477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45234)
0926 20:20:25.829613 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.829711 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 20:20:25.830040 22137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
I20260926 20:20:25.830988 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.830723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45250)
0926 20:20:25.830855 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.830915 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 20:20:25.831167 22188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:38233: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
I20260926 20:20:25.855888 22113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } attempt: 1
I20260926 20:20:25.856907 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.856550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45266)
0926 20:20:25.856750 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.856820 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 20:20:25.857786 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.857542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45278)
0926 20:20:25.857668 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.857722 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 20:20:25.857949 22113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:38233: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
I20260926 20:20:25.880461 22188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } attempt: 1
I20260926 20:20:25.881562 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.881185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45292)
0926 20:20:25.881405 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.881476 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 20:20:25.882478 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.882234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45294)
0926 20:20:25.882364 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.882419 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 20:20:25.882609 22188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:38233: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
I20260926 20:20:25.933220 22113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } attempt: 2
I20260926 20:20:25.934160 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.933855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45310)
0926 20:20:25.934007 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.934077 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 20:20:25.934845 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.934641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45312)
0926 20:20:25.934740 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.934789 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 20:20:25.934964 22113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:38233: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
W20260926 20:20:25.945487 22195 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:20:25.945837 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:25.945895 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:25.945986 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:25.950161 22195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:25.950243 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:25.950276 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:25.950330 22195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:25.950366 22195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:25.950766 22188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } attempt: 2
I20260926 20:20:25.951584 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.951298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45314)
0926 20:20:25.951436 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.951506 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 20:20:25.952359 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:25.952128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45324)
0926 20:20:25.952237 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:25.952301 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:20:25.952481 22188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:38233: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
I20260926 20:20:25.956326 22195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:40169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:37243,127.31.119.253:41281,127.31.119.252:38233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab
--principal=kudu/127.31.119.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.252:38233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.119.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:25.957505 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:25.958819 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:25.968183 22201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:25.968225 22204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:25.968187 22202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:25.968801 22195 server_base.cc:1034] running on GCE node
I20260926 20:20:25.969485 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:25.970702 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:25.971902 22195 hybrid_clock.cc:648] HybridClock initialized: now 1790454025971885 us; error 32 us; skew 500 ppm
Sep 26 20:20:25 dist-test-slave-vnbh krb5kdc[22032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:25.974823 22195 init.cc:377] Logged in from keytab as kudu/127.31.119.252@KRBTEST.COM (short username kudu)
I20260926 20:20:25.975876 22195 webserver.cc:460] Webserver started at http://127.31.119.252:40563/ using document root <none> and password file <none>
I20260926 20:20:25.976486 22195 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:25.976577 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:25.976809 22195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:25.978971 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/data/instance:
uuid: "b955d82fbc42436a8e88e3c7d6484480"
format_stamp: "Formatted at 2026-09-26 20:20:25 on dist-test-slave-vnbh"
server_key: "0c9c8e0ca14e772dbf47cce929a00eae"
server_key_iv: "371e40c79c6b0e11665c7f696f3fec63"
server_key_version: "encryptionkey@0"
I20260926 20:20:25.979575 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/wal/instance:
uuid: "b955d82fbc42436a8e88e3c7d6484480"
format_stamp: "Formatted at 2026-09-26 20:20:25 on dist-test-slave-vnbh"
server_key: "0c9c8e0ca14e772dbf47cce929a00eae"
server_key_iv: "371e40c79c6b0e11665c7f696f3fec63"
server_key_version: "encryptionkey@0"
I20260926 20:20:25.983814 22195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:25.986598 22211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:25.987960 22195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:20:25.988125 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/wal
uuid: "b955d82fbc42436a8e88e3c7d6484480"
format_stamp: "Formatted at 2026-09-26 20:20:25 on dist-test-slave-vnbh"
server_key: "0c9c8e0ca14e772dbf47cce929a00eae"
server_key_iv: "371e40c79c6b0e11665c7f696f3fec63"
server_key_version: "encryptionkey@0"
I20260926 20:20:25.988257 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:26.025811 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:26.029448 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:26.029675 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:26.034132 22113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } attempt: 3
I20260926 20:20:26.035157 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.034706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45326)
0926 20:20:26.034966 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:26.035042 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260926 20:20:26.036059 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.035768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45328)
0926 20:20:26.035899 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:26.035964 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 20:20:26.036234 22113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:38233: Network error: Client connection negotiation failed: client connection to 127.31.119.252:38233: connect: Connection refused (error 111)
I20260926 20:20:26.037624 22195 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.252:38233
I20260926 20:20:26.037920 22263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.252:38233 every 8 connection(s)
I20260926 20:20:26.038661 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
I20260926 20:20:26.041819 22264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:26.043659 22188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } attempt: 3
I20260926 20:20:26.046351 22264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:26.046797 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 22195
I20260926 20:20:26.046933 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453737971229-32223-0/minicluster-data/master-2/wal/instance
I20260926 20:20:26.047230 32223 external_mini_cluster.cc:1428] Setting key 26b6a4268b645d07956de6c3038a2484
Sep 26 20:20:26 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
Sep 26 20:20:26 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
Sep 26 20:20:26 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:20:26.058311 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.044199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45330)
0926 20:20:26.044347 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:26.044367 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:26.044467 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:26.047110 (+ 2643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:26.047113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:26.047320 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:26.047442 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:26.047449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.049317 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.049321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:26.050165 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:26.050176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.051065 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.051069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:26.051149 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.051716 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:26.051734 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:26.053422 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:26.055585 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:26.055589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:26.055591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:26.055852 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:26.056208 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:26.056220 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:26.056222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:26.056311 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:26.057638 (+ 1327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:26.057642 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:26.057842 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 20:20:26.057955 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 20:20:26.058111 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 20:20:26.059525 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.048487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.31.119.254:37243)
0926 20:20:26.048601 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.048605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.048945 (+ 340us) server_negotiation.cc:407] Connection header received
0926 20:20:26.049130 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.049133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.049181 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.049233 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.050025 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.050698 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.051885 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.052172 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.052573 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.055800 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.055805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.055810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.055839 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.057376 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.057998 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.058001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.058003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.058092 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.058430 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.058435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.058438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.058695 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.058776 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.059243 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.059363 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:20:26.059661 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.048381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:37243 (local address 127.0.0.1:49812)
0926 20:20:26.048833 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:26.048878 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:26.049020 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:26.049415 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:26.049419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:26.049746 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:26.049906 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:26.049913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.050828 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.050831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:26.051759 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:26.051772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.052325 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.052330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:26.052449 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.052942 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:26.052961 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:26.055637 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:26.057476 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:26.057486 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:26.057492 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:26.057878 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:26.058204 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:26.058207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:26.058219 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:26.058312 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:26.058859 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:26.058865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:26.059005 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 20:20:26.059162 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 20:20:26.059349 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":50,"spinlock_wait_cycles":2816,"thread_start_us":130,"threads_started":1}
I20260926 20:20:26.059665 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.045060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45330 (local address 127.31.119.252:38233)
0926 20:20:26.045556 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.045571 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.045609 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:20:26.046262 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.046281 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.046709 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.047183 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.047661 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.049165 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.050281 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.050950 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.051109 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.053567 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.053575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.053590 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.053621 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.055459 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.055978 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.055985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.055991 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.056128 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.056426 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.056431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.056435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.057519 (+ 1084us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.057665 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.058109 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.058413 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":181,"threads_started":1}
I20260926 20:20:26.059909 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.047809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.31.119.252:38233)
0926 20:20:26.048278 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.048287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.048301 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:20:26.048465 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.048468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.048519 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.048602 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.049613 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.050412 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.051515 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.051868 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.051972 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.055346 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.055353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.055358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.055389 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.057805 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.058275 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.058279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.058280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.058375 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.058677 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.058680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.058682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.058869 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.059035 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.059526 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.059629 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":398,"spinlock_wait_cycles":2304,"thread_start_us":107,"threads_started":1}
I20260926 20:20:26.061229 22264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:26.064129 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416: Bootstrap starting.
Sep 26 20:20:26 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
I20260926 20:20:26.066994 22188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:26.067896 22188 log.cc:824] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:26.070384 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416: No bootstrap required, opened a new log
Sep 26 20:20:26 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:20:26.071141 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.061734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:41281 (local address 127.0.0.1:43756)
0926 20:20:26.062020 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:26.062031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:26.062128 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:26.062455 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:26.062459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:26.062631 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:26.062732 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:26.062737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.063534 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.063540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:26.064143 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:26.064152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.064615 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.064619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:26.064707 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.065213 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:26.065225 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:26.066920 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:26.069080 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:26.069084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:26.069086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:26.069427 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:26.069779 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:26.069782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:26.069785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:26.069884 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:26.070516 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:26.070521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:26.070677 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 20:20:26.070788 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:20:26.070948 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260926 20:20:26.071695 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.061830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.31.119.253:41281)
0926 20:20:26.061944 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.061953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.062089 (+ 136us) server_negotiation.cc:407] Connection header received
0926 20:20:26.062203 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.062207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.062251 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.062338 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.062856 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.063430 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.064250 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.064500 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.064617 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.067065 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.067068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.067071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.067103 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.068972 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.069558 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.069564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.069565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.069656 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.070003 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.070008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.070010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.070303 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.070448 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.071379 (+ 931us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.071509 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:20:26.072304 22264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:26.074829 22188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } }
I20260926 20:20:26.075166 22188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888b9f66d431411ca1a54cb94480d416, State: Initialized, Role: FOLLOWER
I20260926 20:20:26.075620 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.062476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.31.119.254:37243)
0926 20:20:26.065453 (+ 2977us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.065458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.065474 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:20:26.065515 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.065518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.065563 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.065650 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.066437 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.067052 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.068237 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.068614 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.068736 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.071716 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.071727 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.071730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.071773 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.073884 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.074299 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.074303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.074305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.074394 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.074704 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.074707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.074710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.074916 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.075134 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.075314 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.075438 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2876}
I20260926 20:20:26.075630 22188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } }
I20260926 20:20:26.077229 22188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:26.077551 22271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } } }
I20260926 20:20:26.077728 22271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 20:20:26 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
I20260926 20:20:26.081476 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.072881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45348)
0926 20:20:26.072992 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:26.073008 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:26.073087 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:26.073376 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:26.073379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:26.073537 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:26.073658 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:26.073674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.074650 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.074653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:26.075432 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:26.075442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.076123 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.076126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:26.076209 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.076769 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:26.076780 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:26.078139 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:26.079916 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:26.079922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:26.079925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:26.080145 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:26.080416 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:26.080418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:26.080420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:26.080496 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:26.080904 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:26.080906 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:26.080998 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 20:20:26.081093 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:20:26.081238 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 20:20:26.082271 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.072947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.31.119.252:38233)
0926 20:20:26.073041 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.073046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.073059 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:20:26.073192 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.073197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.073240 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.073358 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.073789 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.074533 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.075551 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.075905 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.076014 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.078268 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.078271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.078273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.078295 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.079816 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.080230 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.080236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.080237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.080319 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.080598 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.080600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.080602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.080786 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.080872 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.081938 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.082071 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 20:20:26 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
I20260926 20:20:26.082299 22120 master_runner.cc:186] Error getting master registration for 127.31.119.254:37243: OK, instance_id { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" instance_seqno: 1790454025561198 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:20:26.085711 22120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:20:26.085693 22264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480: Bootstrap starting.
W20260926 20:20:26.086786 22282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:26.086846 22282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:26.087384 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.077145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43772 (local address 127.31.119.253:41281)
0926 20:20:26.077248 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.077254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.077475 (+ 221us) server_negotiation.cc:407] Connection header received
0926 20:20:26.077650 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.077655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.077817 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.077884 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.078839 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.079432 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.080572 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.080906 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.083820 (+ 2914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.083914 (+ 94us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.083916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.083918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.083931 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.085455 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.085947 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.085951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.085953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.086037 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.086339 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.086342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.086344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.086552 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.086744 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.087081 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.087213 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:20:26.089216 22264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:26.090248 22264 log.cc:824] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:26.090654 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:26.092532 22264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480: No bootstrap required, opened a new log
I20260926 20:20:26.091893 22087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49826
I20260926 20:20:26.096930 22264 raft_consensus.cc:348] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } }
I20260926 20:20:26.097205 22264 raft_consensus.cc:729] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b955d82fbc42436a8e88e3c7d6484480, State: Initialized, Role: FOLLOWER
I20260926 20:20:26.097601 22264 consensus_queue.cc:260] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } }
I20260926 20:20:26.098156 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } } }
I20260926 20:20:26.098308 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:26.099097 22264 sys_catalog.cc:564] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:20:26.104585 22296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:26.104736 22296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:26.102746 22195 master_runner.cc:186] Error getting master registration for 127.31.119.254:37243: OK, instance_id { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" instance_seqno: 1790454025561198 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:20:26.106118 22195 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:20:26.200389 22113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } attempt: 4
Sep 26 20:20:26 dist-test-slave-vnbh krb5kdc[22032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454025, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
I20260926 20:20:26.211027 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.201047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:38233 (local address 127.0.0.1:45354)
0926 20:20:26.201423 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:26.201439 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:26.201541 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:26.201858 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:26.201861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:26.202082 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:26.202196 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:26.202203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.203270 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.203280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:26.204126 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:26.204138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.204875 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.204879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:26.204974 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.205594 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:26.205609 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:26.207249 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:26.209334 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:26.209341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:26.209344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:26.209630 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:26.209927 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:26.209930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:26.209931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:26.210036 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:26.210451 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:26.210456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:26.210570 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 20:20:26.210694 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 20:20:26.210852 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260926 20:20:26.211632 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.201411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.31.119.252:38233)
0926 20:20:26.201581 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.201588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.201610 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:20:26.201662 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.201675 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.201729 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.201842 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.202330 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.203140 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.204266 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.204594 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.204726 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.207402 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.207406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.207409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.207438 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.209180 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.209742 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.209746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.209748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.209832 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.210128 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.210131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.210132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.210351 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.210442 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.211352 (+ 910us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.211490 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:20:26.215075 22113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3: Bootstrap starting.
I20260926 20:20:26.218117 22113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:26.219106 22113 log.cc:824] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:26.221618 22113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3: No bootstrap required, opened a new log
I20260926 20:20:26.224826 22113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } }
I20260926 20:20:26.225073 22113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b3b77dd6334cdaaa93c90e8ae4f1a3, State: Initialized, Role: FOLLOWER
I20260926 20:20:26.225443 22113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } }
I20260926 20:20:26.226086 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } } }
I20260926 20:20:26.226363 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:26.226944 22113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:26.226994 22087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:20:26.227101 22087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:26.228372 22087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } }
I20260926 20:20:26.230737 22162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "888b9f66d431411ca1a54cb94480d416"
I20260926 20:20:26.231027 22162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:26.232385 22162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b3b77dd6334cdaaa93c90e8ae4f1a3 in term 1.
I20260926 20:20:26.232872 22236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b955d82fbc42436a8e88e3c7d6484480"
I20260926 20:20:26.233114 22236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:26.233783 22061 leader_election.cc:302] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77b3b77dd6334cdaaa93c90e8ae4f1a3, 888b9f66d431411ca1a54cb94480d416; no voters:
I20260926 20:20:26.237958 22236 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b3b77dd6334cdaaa93c90e8ae4f1a3 in term 1.
I20260926 20:20:26.229452 22087 leader_election.cc:288] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [CANDIDATE]: Term 1 election: Requested vote from peers 888b9f66d431411ca1a54cb94480d416 (127.31.119.253:41281), b955d82fbc42436a8e88e3c7d6484480 (127.31.119.252:38233)
W20260926 20:20:26.232827 22310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:26.241398 22310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:26.242122 22045 master_runner.cc:263] No leader master found from master 77b3b77dd6334cdaaa93c90e8ae4f1a3
I20260926 20:20:26.242195 22045 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:20:26.244189 22299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:26.244525 22299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [term 1 LEADER]: Becoming Leader. State: Replica: 77b3b77dd6334cdaaa93c90e8ae4f1a3, State: Running, Role: LEADER
I20260926 20:20:26.244975 22299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } }
I20260926 20:20:26.247218 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77b3b77dd6334cdaaa93c90e8ae4f1a3. Latest consensus state: current_term: 1 leader_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } } }
I20260926 20:20:26.247399 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:26.249614 22318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:26.250653 22318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:26.271570 22236 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [term 1 FOLLOWER]: Refusing update from remote peer 77b3b77dd6334cdaaa93c90e8ae4f1a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:26.272500 22316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:26.276122 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.251057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.31.119.254:37243)
0926 20:20:26.251165 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.251169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.254617 (+ 3448us) server_negotiation.cc:407] Connection header received
0926 20:20:26.254848 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.254852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.254904 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.254960 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.256275 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.257065 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.263047 (+ 5982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.263408 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.263500 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.269383 (+ 5883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.269388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.269391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.269424 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.272752 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.273657 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.273661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.273663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.273752 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.274469 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.274472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.274474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.274705 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.274817 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.275814 (+ 997us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.275956 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:20:26.276661 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.249704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.31.119.253:41281)
0926 20:20:26.249827 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.249831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.257536 (+ 7705us) server_negotiation.cc:407] Connection header received
0926 20:20:26.257724 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.257727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.257781 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.257832 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.262918 (+ 5086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.263693 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.264965 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.265344 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.265422 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.267417 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.267421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.267424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.267454 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.269160 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.274056 (+ 4896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.274060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.274062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.274153 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.275129 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.275132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.275134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.275371 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.275486 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.276377 (+ 891us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.276510 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:20:26.280964 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77b3b77dd6334cdaaa93c90e8ae4f1a3. Latest consensus state: current_term: 1 leader_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } } }
I20260926 20:20:26.281111 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:20:26.281155 22217 tablet.cc:2369] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:20:26.282665 22141 tablet.cc:2369] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:20:26.282891 22162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [term 1 FOLLOWER]: Refusing update from remote peer 77b3b77dd6334cdaaa93c90e8ae4f1a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:26.283618 22316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:26.288347 22271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77b3b77dd6334cdaaa93c90e8ae4f1a3. Latest consensus state: current_term: 1 leader_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } } }
I20260926 20:20:26.288501 22271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:26.290437 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:26.251815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.31.119.252:38233)
0926 20:20:26.251938 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.251942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.256188 (+ 4246us) server_negotiation.cc:407] Connection header received
0926 20:20:26.258425 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.258429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.258485 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.258542 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.261416 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.262363 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.266436 (+ 4073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:26.266881 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:26.266972 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:26.269413 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:26.269417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:26.269420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:26.269451 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:26.271818 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.288868 (+ 17050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.288873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.288875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.289013 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:26.289410 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:26.289414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:26.289416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:26.289654 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:26.289784 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:26.290130 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 20:20:26.290270 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:20:26.293931 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } } }
I20260926 20:20:26.294063 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:26.294479 22271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } } }
I20260926 20:20:26.294595 22271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:26.294816 22322 mvcc.cc:205] Tried to move back new op lower bound from 7333699691596947456 to 7333699691504365568. Current Snapshot: MvccSnapshot[applied={T|T < 7333699691596947456}]
I20260926 20:20:26.298558 22325 mvcc.cc:205] Tried to move back new op lower bound from 7333699691596947456 to 7333699691504365568. Current Snapshot: MvccSnapshot[applied={T|T < 7333699691596947456}]
I20260926 20:20:26.301234 22316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } } }
I20260926 20:20:26.301419 22316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:26.302232 22318 catalog_manager.cc:1347] Generated new cluster ID: 1fa4a638a54e40a19069f478a9dc7a38
I20260926 20:20:26.302358 22318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:26.306559 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b3b77dd6334cdaaa93c90e8ae4f1a3" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37243 } } peers { permanent_uuid: "888b9f66d431411ca1a54cb94480d416" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 41281 } } peers { permanent_uuid: "b955d82fbc42436a8e88e3c7d6484480" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 38233 } } }
I20260926 20:20:26.306694 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:26.307319 22323 mvcc.cc:205] Tried to move back new op lower bound from 7333699691596947456 to 7333699691504365568. Current Snapshot: MvccSnapshot[applied={T|T < 7333699691596947456}]
I20260926 20:20:26.332275 22318 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:26.333761 22318 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:26.340812 22318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77b3b77dd6334cdaaa93c90e8ae4f1a3: Generated new TSK 0
I20260926 20:20:26.342054 22318 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:20:26.367023 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.365176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.31.119.253:41281)
0926 20:20:26.365462 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.365469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.365641 (+ 172us) server_negotiation.cc:407] Connection header received
0926 20:20:26.365951 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.365957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.366017 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.366094 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.366868 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41794: BlockingRecv error: recv got EOF from 127.0.0.1:41794 (error 108)
Metrics: {"server-negotiator.queue_time_us":188}
W20260926 20:20:26.367023 22117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.365656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.31.119.254:37243)
0926 20:20:26.365786 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.365790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.365803 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:20:26.365847 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.365849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.365919 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.366012 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.366880 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51758: BlockingRecv error: recv got EOF from 127.0.0.1:51758 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 20:20:26.367352 22266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.365583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.31.119.252:38233)
0926 20:20:26.365726 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.365731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.366158 (+ 427us) server_negotiation.cc:407] Connection header received
0926 20:20:26.366499 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.366502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.366554 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.366619 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.367196 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46844: BlockingRecv error: recv got EOF from 127.0.0.1:46844 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 20:20:26.371173 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.369964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.31.119.253:41281)
0926 20:20:26.370105 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.370109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.370197 (+ 88us) server_negotiation.cc:407] Connection header received
0926 20:20:26.370411 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.370414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.370481 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.370560 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.371047 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41796: BlockingRecv error: recv got EOF from 127.0.0.1:41796 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 20:20:26.375347 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.374473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.31.119.253:41281)
0926 20:20:26.374596 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.374601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.374616 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:20:26.374665 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.374668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.374717 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.374800 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.375235 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41800: BlockingRecv error: recv got EOF from 127.0.0.1:41800 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 20:20:26.380491 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.379190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.31.119.253:41281)
0926 20:20:26.379331 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.379337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.379519 (+ 182us) server_negotiation.cc:407] Connection header received
0926 20:20:26.379683 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.379686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.379733 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.379846 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.380370 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41816: BlockingRecv error: recv got EOF from 127.0.0.1:41816 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 20:20:26.387953 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.386782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.31.119.253:41281)
0926 20:20:26.386955 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.386960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.387031 (+ 71us) server_negotiation.cc:407] Connection header received
0926 20:20:26.387180 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.387184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.387244 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.387350 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.387835 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41818: BlockingRecv error: recv got EOF from 127.0.0.1:41818 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260926 20:20:26.399789 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.398313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.31.119.253:41281)
0926 20:20:26.398462 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.398467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.398611 (+ 144us) server_negotiation.cc:407] Connection header received
0926 20:20:26.398811 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.398815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.398873 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.398959 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.399642 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41830: BlockingRecv error: recv got EOF from 127.0.0.1:41830 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 20:20:26.419384 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.417996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.31.119.253:41281)
0926 20:20:26.418153 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.418159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.418366 (+ 207us) server_negotiation.cc:407] Connection header received
0926 20:20:26.418565 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.418568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.418627 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.418726 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.419263 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41838: BlockingRecv error: recv got EOF from 127.0.0.1:41838 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 20:20:26.455024 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.453692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.31.119.253:41281)
0926 20:20:26.453886 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.453890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.453970 (+ 80us) server_negotiation.cc:407] Connection header received
0926 20:20:26.454101 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.454104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.454156 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.454233 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.454891 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41842: BlockingRecv error: recv got EOF from 127.0.0.1:41842 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 20:20:26.522704 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.521286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.31.119.253:41281)
0926 20:20:26.521451 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.521458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.521573 (+ 115us) server_negotiation.cc:407] Connection header received
0926 20:20:26.521865 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.521869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.521922 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.522024 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.522548 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41844: BlockingRecv error: recv got EOF from 127.0.0.1:41844 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 20:20:26.654417 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.653141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.31.119.253:41281)
0926 20:20:26.653308 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.653312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.653504 (+ 192us) server_negotiation.cc:407] Connection header received
0926 20:20:26.653593 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.653596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.653649 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.653746 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.654287 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41860: BlockingRecv error: recv got EOF from 127.0.0.1:41860 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 20:20:26.914002 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:26.912846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.31.119.253:41281)
0926 20:20:26.913016 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:26.913027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:26.913043 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:20:26.913099 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:26.913101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:26.913153 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:26.913240 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:26.913878 (+ 638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41866: BlockingRecv error: recv got EOF from 127.0.0.1:41866 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:20:27.088223 22282 catalog_manager.cc:1259] Loaded cluster ID: 1fa4a638a54e40a19069f478a9dc7a38
I20260926 20:20:27.088325 22282 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416: loading cluster ID for follower catalog manager: success
I20260926 20:20:27.090246 22282 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416: acquiring CA information for follower catalog manager: success
I20260926 20:20:27.091683 22282 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 888b9f66d431411ca1a54cb94480d416: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:20:27.106210 22296 catalog_manager.cc:1259] Loaded cluster ID: 1fa4a638a54e40a19069f478a9dc7a38
I20260926 20:20:27.106328 22296 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480: loading cluster ID for follower catalog manager: success
I20260926 20:20:27.107987 22296 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480: acquiring CA information for follower catalog manager: success
I20260926 20:20:27.109135 22296 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b955d82fbc42436a8e88e3c7d6484480: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:20:27.432358 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:27.428248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.31.119.253:41281)
0926 20:20:27.428534 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:27.428537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:27.428555 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:20:27.428686 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:27.428689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:27.428816 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:27.428929 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:27.429310 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:27.429899 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:27.430789 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:27.431014 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:27.431098 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:27.431450 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:27.431908 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:27.432003 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:27.432086 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 20:20:27.432139 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":102,"threads_started":1}
I20260926 20:20:27.439615 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:27.435688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.31.119.252:38233)
0926 20:20:27.435918 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:27.435922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:27.435935 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:20:27.436029 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:27.436032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:27.436149 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:27.436248 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:27.436476 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:27.437015 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:27.438016 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:27.438250 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:27.438342 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:27.438613 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:27.439171 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:27.439280 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:27.439362 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 20:20:27.439416 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260926 20:20:27.447350 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:27.443576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.31.119.252:38233)
0926 20:20:27.443696 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:27.443701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:27.443859 (+ 158us) server_negotiation.cc:407] Connection header received
0926 20:20:27.443975 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:27.443985 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:27.444035 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:27.444196 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:27.444551 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:27.445267 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:27.446146 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:27.446462 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:27.446557 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:27.446830 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:27.446930 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:27.447003 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:27.447160 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 20:20:27.447213 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:20:27.447537 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:27.443086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.31.119.253:41281)
0926 20:20:27.443343 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:27.443349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:27.443517 (+ 168us) server_negotiation.cc:407] Connection header received
0926 20:20:27.443701 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:27.443704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:27.443757 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:27.443807 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:27.444173 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:27.444941 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:27.446287 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:27.446633 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:27.446727 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:27.447061 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:27.447150 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:27.447212 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:27.447367 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 20:20:27.447429 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260926 20:20:27.447968 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:27.443053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.31.119.254:37243)
0926 20:20:27.443508 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:27.443512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:27.443526 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:20:27.443621 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:27.443625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:27.443791 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:27.443905 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:27.444251 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:27.444916 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:27.446309 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:27.446563 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:27.446664 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:27.446997 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:27.447371 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:27.447461 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:27.447630 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:20:27.447697 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":73,"threads_started":1}
I20260926 20:20:27.458042 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 22045
I20260926 20:20:27.464848 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 22120
I20260926 20:20:27.471024 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 22195
2026-09-26T20:20:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3272 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:27 dist-test-slave-vnbh krb5kdc[22412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:27 dist-test-slave-vnbh krb5kdc[22412](info): set up 2 sockets
Sep 26 20:20:27 dist-test-slave-vnbh krb5kdc[22412](info): commencing operation
krb5kdc: starting...
W20260926 20:20:28.514801 32223 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/test-user.keytab.
Sep 26 20:20:28 dist-test-slave-vnbh krb5kdc[22412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.119.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
Entry for principal kudu/127.31.119.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab.
I20260926 20:20:28.648473 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:37717
--webserver_interface=127.31.119.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:37717,127.31.119.253:40849,127.31.119.252:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:28.762466 22425 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:20:28.762781 22425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:28.762806 22425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:28.762846 22425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:28.766886 22425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:28.766955 22425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:28.766975 22425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:28.766992 22425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:28.767007 22425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:28.772586 22425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:37717,127.31.119.253:40849,127.31.119.252:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.254.keytab
--principal=kudu/127.31.119.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.254:37717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.119.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:28.773741 22425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:28.774814 22425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:28.783780 22430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:28.783728 22433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:28.783747 22431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:28.784416 22425 server_base.cc:1034] running on GCE node
I20260926 20:20:28.785075 22425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:28.786338 22425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:28.787509 22425 hybrid_clock.cc:648] HybridClock initialized: now 1790454028787486 us; error 40 us; skew 500 ppm
Sep 26 20:20:28 dist-test-slave-vnbh krb5kdc[22412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:28.790686 22425 init.cc:377] Logged in from keytab as kudu/127.31.119.254@KRBTEST.COM (short username kudu)
I20260926 20:20:28.791764 22425 webserver.cc:460] Webserver started at http://127.31.119.254:33579/ using document root <none> and password file <none>
I20260926 20:20:28.792397 22425 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:28.792491 22425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:28.792724 22425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:28.795055 22425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/data/instance:
uuid: "bf36e47a4a8b42eeb0928d0f58180602"
format_stamp: "Formatted at 2026-09-26 20:20:28 on dist-test-slave-vnbh"
server_key: "85ce064b80408a00b09d238a274ad671"
server_key_iv: "d9f5cfc9eb0f7d526027d50fd5dc5f54"
server_key_version: "encryptionkey@0"
I20260926 20:20:28.795680 22425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/wal/instance:
uuid: "bf36e47a4a8b42eeb0928d0f58180602"
format_stamp: "Formatted at 2026-09-26 20:20:28 on dist-test-slave-vnbh"
server_key: "85ce064b80408a00b09d238a274ad671"
server_key_iv: "d9f5cfc9eb0f7d526027d50fd5dc5f54"
server_key_version: "encryptionkey@0"
I20260926 20:20:28.799841 22425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 20:20:28.802539 22440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:28.803907 22425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:20:28.804068 22425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/wal
uuid: "bf36e47a4a8b42eeb0928d0f58180602"
format_stamp: "Formatted at 2026-09-26 20:20:28 on dist-test-slave-vnbh"
server_key: "85ce064b80408a00b09d238a274ad671"
server_key_iv: "d9f5cfc9eb0f7d526027d50fd5dc5f54"
server_key_version: "encryptionkey@0"
I20260926 20:20:28.804210 22425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:28.819275 22425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:28.822538 22425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:28.822746 22425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:28.829897 22425 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.254:37717
I20260926 20:20:28.830169 22492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.254:37717 every 8 connection(s)
I20260926 20:20:28.830876 22425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:28.833699 22493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:28.835424 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 22425
I20260926 20:20:28.835559 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-0/wal/instance
I20260926 20:20:28.835846 32223 external_mini_cluster.cc:1428] Setting key afe42c61aa6aa02a9ab709a00d60fc5b
I20260926 20:20:28.837875 22493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.119.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.253@KRBTEST.COM" created.
Sep 26 20:20:28 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:20:28.855093 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:28.841667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.31.119.254:37717)
0926 20:20:28.841990 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:28.842020 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:28.842071 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:20:28.842627 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:28.842633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:28.842868 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:28.843068 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:28.844002 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:28.844999 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:28.846200 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:28.846701 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:28.846882 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:28.850169 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:28.850175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:28.850186 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:28.850207 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:28.851883 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:28.852374 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:28.852385 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:28.852386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:28.852487 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:28.852816 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:28.852819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:28.852821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:28.853227 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:28.853410 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:28.853942 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 20:20:28.854193 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
I20260926 20:20:28.855116 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:28.840562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:37717 (local address 127.0.0.1:36362)
0926 20:20:28.841141 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:28.841188 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:28.842298 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:28.843063 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:28.843069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:28.843537 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:28.843811 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:28.843826 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:28.845127 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:28.845130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:28.846085 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:28.846095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:28.846901 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:28.846907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:28.847070 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:28.847759 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:28.847788 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:28.850020 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:28.852011 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:28.852021 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:28.852031 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:28.852262 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:28.852578 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:28.852583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:28.852585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:28.852714 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:28.853428 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:28.853437 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:28.853634 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 20:20:28.853851 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 20:20:28.854305 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":73,"thread_start_us":163,"threads_started":1}
I20260926 20:20:28.858069 22493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:28.859046 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:28.858603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:40849 (local address 127.0.0.1:50290)
0926 20:20:28.858896 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:28.858975 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
W20260926 20:20:28.859325 22443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.119.253:40849: connect: Connection refused (error 111)
I20260926 20:20:28.860387 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:28.860184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:40849 (local address 127.0.0.1:50304)
0926 20:20:28.860286 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:28.860332 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:20:28.860584 22493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.253:40849: Network error: Client connection negotiation failed: client connection to 127.31.119.253:40849: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab.
Entry for principal kudu/127.31.119.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab.
I20260926 20:20:28.866479 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.253:40849
--webserver_interface=127.31.119.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab
--principal=kudu/127.31.119.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:37717,127.31.119.253:40849,127.31.119.252:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:20:28.909888 22493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } attempt: 1
I20260926 20:20:28.910868 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:28.910500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:40849 (local address 127.0.0.1:50306)
0926 20:20:28.910721 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:28.910786 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 20:20:28.911731 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:28.911448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:40849 (local address 127.0.0.1:50322)
0926 20:20:28.911580 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:28.911669 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 20:20:28.911897 22493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.253:40849: Network error: Client connection negotiation failed: client connection to 127.31.119.253:40849: connect: Connection refused (error 111)
I20260926 20:20:28.980096 22493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } attempt: 2
W20260926 20:20:28.980571 22500 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:20:28.980957 22500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:28.981009 22500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260926 20:20:28.981032 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:28.980660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:40849 (local address 127.0.0.1:50330)
0926 20:20:28.980823 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:28.980908 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 20:20:28.981096 22500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 20:20:28.981895 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:28.981687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:40849 (local address 127.0.0.1:50342)
0926 20:20:28.981792 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:28.981840 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.253:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 20:20:28.982033 22493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.253:40849: Network error: Client connection negotiation failed: client connection to 127.31.119.253:40849: connect: Connection refused (error 111)
W20260926 20:20:28.985832 22500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:28.985899 22500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:28.985920 22500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:28.985937 22500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:28.985954 22500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:28.991708 22500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:37717,127.31.119.253:40849,127.31.119.252:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.253.keytab
--principal=kudu/127.31.119.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.253:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.119.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:28.993023 22500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:28.994232 22500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:20:29.003607 22500 server_base.cc:1034] running on GCE node
W20260926 20:20:29.003558 22506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:29.003543 22505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:29.003547 22508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:29.004580 22500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:29.005888 22500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:29.007088 22500 hybrid_clock.cc:648] HybridClock initialized: now 1790454029007052 us; error 57 us; skew 500 ppm
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454029, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:29.010751 22500 init.cc:377] Logged in from keytab as kudu/127.31.119.253@KRBTEST.COM (short username kudu)
I20260926 20:20:29.012058 22500 webserver.cc:460] Webserver started at http://127.31.119.253:34327/ using document root <none> and password file <none>
I20260926 20:20:29.012925 22500 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:29.013137 22500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:29.013450 22500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:29.015758 22500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/data/instance:
uuid: "1fc5f08f36a24676a209c97ab19b3398"
format_stamp: "Formatted at 2026-09-26 20:20:29 on dist-test-slave-vnbh"
server_key: "505dd4c3db9408b5e92b71748b4227c0"
server_key_iv: "db685fb211e12a9d418114b2a70c620b"
server_key_version: "encryptionkey@0"
I20260926 20:20:29.016389 22500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/wal/instance:
uuid: "1fc5f08f36a24676a209c97ab19b3398"
format_stamp: "Formatted at 2026-09-26 20:20:29 on dist-test-slave-vnbh"
server_key: "505dd4c3db9408b5e92b71748b4227c0"
server_key_iv: "db685fb211e12a9d418114b2a70c620b"
server_key_version: "encryptionkey@0"
I20260926 20:20:29.020498 22500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:29.023335 22515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:29.024828 22500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:20:29.024999 22500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/wal
uuid: "1fc5f08f36a24676a209c97ab19b3398"
format_stamp: "Formatted at 2026-09-26 20:20:29 on dist-test-slave-vnbh"
server_key: "505dd4c3db9408b5e92b71748b4227c0"
server_key_iv: "db685fb211e12a9d418114b2a70c620b"
server_key_version: "encryptionkey@0"
I20260926 20:20:29.025123 22500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:29.048272 22500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:29.051379 22500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:29.051600 22500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:29.059412 22500 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.253:40849
I20260926 20:20:29.059734 22567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.253:40849 every 8 connection(s)
I20260926 20:20:29.060534 22500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/data/info.pb
I20260926 20:20:29.063563 22568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:29.064050 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 22500
I20260926 20:20:29.064194 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-1/wal/instance
I20260926 20:20:29.064513 32223 external_mini_cluster.cc:1428] Setting key 7a77fee9f1be229fc3015b5ea1680dea
I20260926 20:20:29.068396 22568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.119.252@KRBTEST.COM; defaulting to no policy
I20260926 20:20:29.073210 22493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.119.252@KRBTEST.COM" created.
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454029, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
I20260926 20:20:29.085317 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.073714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:40849 (local address 127.0.0.1:50356)
0926 20:20:29.073885 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.074033 (+ 148us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:29.074117 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:29.074874 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:29.074877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:29.075062 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:29.075167 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.075175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.076682 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.076689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.077652 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.077663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.078426 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.078430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.078546 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.079006 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:29.079026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:29.080880 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:29.083096 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.083100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.083102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.083399 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.083791 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.083795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.083797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.083886 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.084710 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:29.084713 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:29.084823 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 20:20:29.084937 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:20:29.085099 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":1}
I20260926 20:20:29.086228 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.070635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.31.119.254:37717)
0926 20:20:29.070775 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.070779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.071947 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 20:20:29.072952 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.072956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.073022 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.073128 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.074844 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.075833 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.077139 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.077538 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.077658 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.080748 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.080753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.080757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.080791 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.083683 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.084327 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.084331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.084333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.084410 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.084775 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.084781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.084782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.085032 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.085122 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.085956 (+ 834us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.086083 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":46}
I20260926 20:20:29.086643 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.071218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:37717 (local address 127.0.0.1:36368)
0926 20:20:29.071775 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.071834 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:29.072737 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:29.073371 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:29.073381 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:29.073920 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:29.074631 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.074652 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.075952 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.075956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.077019 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.077027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.077719 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.077726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.077944 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.078846 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:29.078874 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:29.080540 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:29.083869 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.083880 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.083889 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.084205 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.084513 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.084519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.084521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.084635 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.085208 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:29.085215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:29.085704 (+ 489us) client_negotiation.cc:769] Sending connection context
0926 20:20:29.085858 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 20:20:29.086013 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":50,"thread_start_us":167,"threads_started":1}
I20260926 20:20:29.086643 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.073833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.31.119.253:40849)
0926 20:20:29.074257 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.074268 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.074292 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:20:29.074333 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.074338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.074528 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.074699 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.075307 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.076554 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.077792 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.078305 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.078419 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.081028 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.081034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.081047 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.081069 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.082971 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.083520 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.083537 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.083543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.083684 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.083988 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.083991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.083992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.084602 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.084750 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.085116 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.085527 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"spinlock_wait_cycles":18304,"thread_start_us":125,"threads_started":1}
I20260926 20:20:29.088074 22568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:29.089188 22493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:29.089990 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.089675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43094)
0926 20:20:29.089844 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.089902 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:20:29.090665 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.090423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43098)
0926 20:20:29.090564 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.090614 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 20:20:29.090777 22493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:36185: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454029, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.119.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab.
Entry for principal kudu/127.31.119.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab.
I20260926 20:20:29.096697 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.088602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:40849 (local address 127.0.0.1:50362)
0926 20:20:29.088735 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.088750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:29.088831 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:29.089340 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:29.089346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:29.089515 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:29.089614 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.089619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.090594 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.090597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.091228 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.091241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.091658 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.091662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.091755 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.092188 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:29.092205 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:29.093585 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:29.095154 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.095160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.095163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.095378 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.095654 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.095656 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.095658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.095761 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.096149 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:29.096152 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:29.096255 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 20:20:29.096374 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 20:20:29.096528 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 20:20:29.097044 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.088681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.31.119.253:40849)
0926 20:20:29.088781 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.088785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.088798 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:20:29.089126 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.089132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.089181 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.089262 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.089785 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.090473 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.091332 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.091556 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.091662 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.093728 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.093731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.093733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.093791 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.095045 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.095472 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.095478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.095480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.095561 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.095876 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.095878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.095879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.096051 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.096149 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.096429 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.096921 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":378,"server-negotiator.queue_time_us":38}
I20260926 20:20:29.097533 32223 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
/tmp/dist-test-taskYRmynJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.252:36185
--webserver_interface=127.31.119.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab
--principal=kudu/127.31.119.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.119.212:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.119.254:37717,127.31.119.253:40849,127.31.119.252:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:20:29.097779 22568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:29.098493 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.098237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43112)
0926 20:20:29.098348 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.098441 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 20:20:29.099216 22516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
I20260926 20:20:29.100283 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.099989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43120)
0926 20:20:29.100134 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.100199 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:20:29.100464 22568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:36185: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
I20260926 20:20:29.121935 22493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } attempt: 1
I20260926 20:20:29.122771 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.122482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43122)
0926 20:20:29.122627 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.122695 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:20:29.123533 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.123347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43136)
0926 20:20:29.123441 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.123485 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 20:20:29.123678 22493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:36185: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
I20260926 20:20:29.149760 22568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } attempt: 1
I20260926 20:20:29.150753 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.150358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43142)
0926 20:20:29.150550 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.150644 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 20:20:29.151505 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.151270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43150)
0926 20:20:29.151389 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.151442 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 20:20:29.151638 22568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:36185: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
I20260926 20:20:29.198871 22493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } attempt: 2
I20260926 20:20:29.199748 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.199446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43152)
0926 20:20:29.199597 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.199667 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:20:29.200529 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.200333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43168)
0926 20:20:29.200430 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.200475 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 20:20:29.200690 22493 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:36185: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
W20260926 20:20:29.214253 22574 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:20:29.214564 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:29.214629 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:29.214715 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:29.219000 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:29.219074 22574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:29.219095 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:29.219134 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:29.219168 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:29.219801 22568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } attempt: 2
I20260926 20:20:29.220674 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.220332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43180)
0926 20:20:29.220507 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.220579 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 20:20:29.221392 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.221148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43182)
0926 20:20:29.221259 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.221332 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 20:20:29.221536 22568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.119.252:36185: Network error: Client connection negotiation failed: client connection to 127.31.119.252:36185: connect: Connection refused (error 111)
I20260926 20:20:29.225268 22574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.119.212:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.119.254:37717,127.31.119.253:40849,127.31.119.252:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/krb5kdc/kudu_127.31.119.252.keytab
--principal=kudu/127.31.119.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.119.252:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.119.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:29.226464 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:29.227528 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:29.236305 22580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:29.236315 22583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:29.236380 22574 server_base.cc:1034] running on GCE node
W20260926 20:20:29.236327 22581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:29.237075 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:29.238312 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:29.239498 22574 hybrid_clock.cc:648] HybridClock initialized: now 1790454029239470 us; error 45 us; skew 500 ppm
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454029, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:29.242435 22574 init.cc:377] Logged in from keytab as kudu/127.31.119.252@KRBTEST.COM (short username kudu)
I20260926 20:20:29.243345 22574 webserver.cc:460] Webserver started at http://127.31.119.252:36509/ using document root <none> and password file <none>
I20260926 20:20:29.243933 22574 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:29.243997 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:29.244231 22574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:29.246465 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/data/instance:
uuid: "cf962ad56f654f52b9eca7149ef973d4"
format_stamp: "Formatted at 2026-09-26 20:20:29 on dist-test-slave-vnbh"
server_key: "1eaa230da3d734472c950efd94a52e4e"
server_key_iv: "c9cb53396f63d0e8faf754b57174b973"
server_key_version: "encryptionkey@0"
I20260926 20:20:29.247053 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/wal/instance:
uuid: "cf962ad56f654f52b9eca7149ef973d4"
format_stamp: "Formatted at 2026-09-26 20:20:29 on dist-test-slave-vnbh"
server_key: "1eaa230da3d734472c950efd94a52e4e"
server_key_iv: "c9cb53396f63d0e8faf754b57174b973"
server_key_version: "encryptionkey@0"
I20260926 20:20:29.250870 22574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:29.253364 22590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:29.254627 22574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:20:29.254758 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/data,/tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/wal
uuid: "cf962ad56f654f52b9eca7149ef973d4"
format_stamp: "Formatted at 2026-09-26 20:20:29 on dist-test-slave-vnbh"
server_key: "1eaa230da3d734472c950efd94a52e4e"
server_key_iv: "c9cb53396f63d0e8faf754b57174b973"
server_key_version: "encryptionkey@0"
I20260926 20:20:29.254875 22574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:29.270032 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:29.273249 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:29.273499 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:29.280930 22574 rpc_server.cc:307] RPC server started. Bound to: 127.31.119.252:36185
I20260926 20:20:29.281267 22642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.119.252:36185 every 8 connection(s)
I20260926 20:20:29.282029 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/data/info.pb
I20260926 20:20:29.284318 32223 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu as pid 22574
I20260926 20:20:29.284469 32223 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYRmynJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453737971229-32223-0/minicluster-data/master-2/wal/instance
I20260926 20:20:29.284782 32223 external_mini_cluster.cc:1428] Setting key 3480092789fd1e6d06bf24d7be8f0464
I20260926 20:20:29.285423 22643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:29.290401 22643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454029, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:20:29.299860 22493 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } attempt: 3
I20260926 20:20:29.302649 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.286362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.31.119.252:36185)
0926 20:20:29.286878 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.286903 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.286947 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:20:29.287728 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.287747 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.288164 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.288556 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.289625 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.290778 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.291759 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.292498 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.292687 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.296591 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.296599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.296614 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.296649 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.298523 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.299002 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.299009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.299016 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.299147 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.299778 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.299781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.299783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.300421 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.300627 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.301132 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.301499 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":150,"threads_started":1}
I20260926 20:20:29.303140 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.292414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.254:37717 (local address 127.0.0.1:36380)
0926 20:20:29.292847 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.292905 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:29.293044 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:29.293350 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:29.293355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:29.293718 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:29.293838 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.293843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.294936 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.294939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.295973 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.295983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.296591 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.296596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.296721 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.297458 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:29.297477 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:29.299160 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:29.301081 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.301091 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.301098 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.301554 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.301847 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.301853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.301855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.301966 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.302388 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:29.302393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:29.302505 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 20:20:29.302645 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 20:20:29.302822 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":158,"threads_started":1}
I20260926 20:20:29.303288 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.292448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.31.119.254:37717)
0926 20:20:29.292585 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.292591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.293018 (+ 427us) server_negotiation.cc:407] Connection header received
0926 20:20:29.293136 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.293139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.293200 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.293296 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.293960 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.294790 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.296075 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.296460 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.296555 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.299293 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.299296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.299298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.299323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.300969 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.301661 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.301664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.301666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.301759 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.302075 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.302079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.302080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.302268 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.302332 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.303012 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.303121 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:20:29.304345 22643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.254@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454029, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
I20260926 20:20:29.310983 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.300298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43192)
0926 20:20:29.300416 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.300431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:29.300511 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:29.301154 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:29.301157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:29.301379 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:29.301495 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.301502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.302536 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.302540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.303336 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.303346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.304088 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.304091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.304177 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.305014 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:29.305028 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:29.306556 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:29.309383 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.309386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.309388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.309615 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.309908 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.309911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.309913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.310008 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.310492 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:29.310496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:29.310605 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 20:20:29.310711 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 20:20:29.310823 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:20:29.311147 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.300581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.31.119.252:36185)
0926 20:20:29.300904 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.300911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.300922 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:20:29.300971 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.300973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.301015 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.301603 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.301686 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.302425 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.303657 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.303978 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.304089 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.306667 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.306670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.306673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.306695 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.309233 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.309707 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.309711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.309713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.309811 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.310115 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.310118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.310120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.310377 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.310486 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.310781 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.310904 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":148,"threads_started":1}
I20260926 20:20:29.312687 22568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } attempt: 3
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.254@KRBTEST.COM
I20260926 20:20:29.314482 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.304974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.31.119.253:40849)
0926 20:20:29.305234 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.305237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.305248 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:20:29.305327 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.305330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.305379 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.305461 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.305867 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.306477 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.307654 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.308011 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.308138 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.310656 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.310659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.310661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.310712 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.312529 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.313060 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.313064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.313066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.313161 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.313525 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.313531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.313535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.313779 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.313850 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.314181 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.314306 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260926 20:20:29.315217 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.304866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.253:40849 (local address 127.0.0.1:50366)
0926 20:20:29.305007 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.305023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:29.305100 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:29.305474 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:29.305477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:29.305646 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:29.305730 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.305736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.306613 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.306617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.307499 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.307510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.308132 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.308135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.308230 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.308830 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:29.308845 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:29.310516 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:29.312634 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.312636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.312638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.312954 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.313275 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.313278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.313303 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.313397 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.313924 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:29.313926 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:29.314014 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 20:20:29.314935 (+ 921us) client_negotiation.cc:240] Negotiation successful
0926 20:20:29.315077 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 20:20:29.315933 22493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602: Bootstrap starting.
I20260926 20:20:29.316107 22643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:20:29.317782 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.306382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.31.119.254:37717)
0926 20:20:29.306511 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.306516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.306530 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:20:29.306680 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.306683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.306723 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.306810 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.307568 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.308379 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.309648 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.309914 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.310004 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.313913 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.313917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.313920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.313949 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.315708 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.316232 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.316248 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.316250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.316340 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.316648 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.316651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.316653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.316851 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.317043 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.317475 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.317597 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454029, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.253@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
I20260926 20:20:29.319607 22493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:29.320565 22493 log.cc:824] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454029, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.119.252@KRBTEST.COM for kudu/127.31.119.252@KRBTEST.COM
I20260926 20:20:29.323454 22493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602: No bootstrap required, opened a new log
I20260926 20:20:29.324680 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.313111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43208)
0926 20:20:29.313240 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.313256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:29.313483 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:29.313897 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:29.313900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:29.314082 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:29.314533 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.314545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.316006 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.316013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.316840 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.316849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.317814 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.317821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.317922 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.318414 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:29.318428 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:29.320322 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:29.323079 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.323082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.323084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.323313 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.323627 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.323631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.323633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.323745 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.324143 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:29.324145 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:29.324243 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 20:20:29.324368 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 20:20:29.324505 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 20:20:29.325136 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.313387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.31.119.252:36185)
0926 20:20:29.313658 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.313663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.313677 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:20:29.313712 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.313715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.313770 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.314660 (+ 890us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.315047 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.315882 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.317319 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.317665 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.321398 (+ 3733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.321504 (+ 106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.321507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.321509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.321532 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.322959 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.323397 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.323401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.323403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.323494 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.323851 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.323856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.323857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.324046 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.324151 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.324427 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.324577 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
Sep 26 20:20:29 dist-test-slave-vnbh krb5kdc[22412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.119.253@KRBTEST.COM
I20260926 20:20:29.328714 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.316831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.31.119.252:36185)
0926 20:20:29.317001 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.317007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.317020 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:20:29.317059 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.317064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.317116 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.317191 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.317604 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.318593 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.319571 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.319892 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.319988 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.324704 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.324707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.324710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.324728 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.326474 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.326909 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.326912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.326913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.326976 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.327208 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.327210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.327212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.327411 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.327505 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.328466 (+ 961us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.328579 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 20:20:29.328915 22568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398: Bootstrap starting.
I20260926 20:20:29.328752 22493 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } }
I20260926 20:20:29.328990 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.316637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.119.252:36185 (local address 127.0.0.1:43220)
0926 20:20:29.316915 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:29.316928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:29.316996 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:29.317220 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:29.317223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:29.317413 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:29.317494 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.317500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.318694 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.318698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.319456 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:29.319467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.320096 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.320099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:29.320173 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.320661 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:29.320679 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:29.324577 (+ 3898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:29.326588 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.326591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.326592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.326812 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.327055 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:29.327058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:29.327059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:29.327122 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:29.327538 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:29.327540 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:29.327615 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 20:20:29.327989 (+ 374us) client_negotiation.cc:240] Negotiation successful
0926 20:20:29.328121 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222}
I20260926 20:20:29.329015 22493 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf36e47a4a8b42eeb0928d0f58180602, State: Initialized, Role: FOLLOWER
I20260926 20:20:29.329517 22493 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } }
I20260926 20:20:29.331030 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.319936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.31.119.253:40849)
0926 20:20:29.320038 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.320042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.320094 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:20:29.320272 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.320275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.320331 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.320688 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.321611 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.322337 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.323342 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.323667 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.323781 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.327400 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.327403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.327406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.327431 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.329112 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.329639 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.329644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.329649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.329740 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.330038 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.330043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.330048 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.330261 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.330439 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.330785 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.330896 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:20:29.331013 22493 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:29.332298 22643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4: Bootstrap starting.
I20260926 20:20:29.332909 22568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:29.335844 22643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:29.336306 22568 log.cc:824] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398: Log is configured to *not* fsync() on all Append() calls
W20260926 20:20:29.336432 22661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:29.336493 22661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:29.336627 32223 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:29.337455 22643 log.cc:824] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:29.337913 22467 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bf36e47a4a8b42eeb0928d0f58180602"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36390
I20260926 20:20:29.338141 22467 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:20:29.338270 22467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:29.339457 22467 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } }
I20260926 20:20:29.336519 22425 master_runner.cc:186] Error getting master registration for 127.31.119.253:40849: OK, instance_id { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" instance_seqno: 1790454029057088 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:20:29.340023 22425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:20:29.333709 22650 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } } }
I20260926 20:20:29.340121 22650 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:29.340524 22467 leader_election.cc:288] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [CANDIDATE]: Term 1 election: Requested vote from peers 1fc5f08f36a24676a209c97ab19b3398 (127.31.119.253:40849), cf962ad56f654f52b9eca7149ef973d4 (127.31.119.252:36185)
I20260926 20:20:29.344365 22568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398: No bootstrap required, opened a new log
I20260926 20:20:29.348949 22568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } }
I20260926 20:20:29.349236 22568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc5f08f36a24676a209c97ab19b3398, State: Initialized, Role: FOLLOWER
I20260926 20:20:29.349644 22568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } }
I20260926 20:20:29.351227 22568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:29.351259 22542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf36e47a4a8b42eeb0928d0f58180602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1fc5f08f36a24676a209c97ab19b3398"
I20260926 20:20:29.351446 22643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4: No bootstrap required, opened a new log
I20260926 20:20:29.351886 22542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:29.353423 22542 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf36e47a4a8b42eeb0928d0f58180602 in term 1.
I20260926 20:20:29.354060 22443 leader_election.cc:302] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc5f08f36a24676a209c97ab19b3398, bf36e47a4a8b42eeb0928d0f58180602; no voters:
I20260926 20:20:29.354453 22650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:29.354694 22650 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [term 1 LEADER]: Becoming Leader. State: Replica: bf36e47a4a8b42eeb0928d0f58180602, State: Running, Role: LEADER
I20260926 20:20:29.355069 22650 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } }
I20260926 20:20:29.355866 22643 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } }
I20260926 20:20:29.356130 22643 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf962ad56f654f52b9eca7149ef973d4, State: Initialized, Role: FOLLOWER
I20260926 20:20:29.356464 22643 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } }
I20260926 20:20:29.357012 22650 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf36e47a4a8b42eeb0928d0f58180602. Latest consensus state: current_term: 1 leader_uuid: "bf36e47a4a8b42eeb0928d0f58180602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } } }
I20260926 20:20:29.357129 22650 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:29.357915 22643 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:29.358173 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.344350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.31.119.252:36185)
0926 20:20:29.344728 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.344733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.344949 (+ 216us) server_negotiation.cc:407] Connection header received
0926 20:20:29.345136 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.345141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.345192 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.345245 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.346198 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.347002 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.348215 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.348549 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.348628 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.350587 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.350595 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.350597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.350622 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.352485 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.352994 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.352998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.353000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.353083 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.353410 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.353413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.353415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.353632 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.353745 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.357879 (+ 4134us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.358018 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282}
I20260926 20:20:29.358912 22617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf36e47a4a8b42eeb0928d0f58180602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cf962ad56f654f52b9eca7149ef973d4"
I20260926 20:20:29.359138 22617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:29.360580 22617 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf36e47a4a8b42eeb0928d0f58180602 in term 1.
I20260926 20:20:29.361999 22687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:29.363101 22687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:29.364933 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.343729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.31.119.254:37717)
0926 20:20:29.343980 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.343985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.344000 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:20:29.344041 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.344044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.344101 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.344186 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.346230 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.346973 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.349757 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.350083 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.350162 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.356444 (+ 6282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.356448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.356451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.356483 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.358586 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.363466 (+ 4880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.363471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.363473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.363572 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.363893 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.363896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.363898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.364138 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.364266 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.364639 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.364770 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260926 20:20:29.366472 22685 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } } }
I20260926 20:20:29.366611 22685 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:20:29.367669 22683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:29.367777 22683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:29.367967 22672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } } }
I20260926 20:20:29.368093 22672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:29.368944 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:29.343845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.31.119.253:40849)
0926 20:20:29.343968 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.343973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.351046 (+ 7073us) server_negotiation.cc:407] Connection header received
0926 20:20:29.351630 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.351633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.351686 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.351752 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.359923 (+ 8171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.360722 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.362060 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:29.362346 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:29.362424 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:29.365044 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:29.365048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:29.365052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:29.365085 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:29.366807 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.367284 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.367288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.367290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.367377 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:29.367667 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:29.367670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:29.367672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:29.367966 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:29.368092 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:29.368669 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 20:20:29.368796 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 20:20:29.370261 22698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:29.370342 22698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:29.372759 22617 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [term 1 FOLLOWER]: Refusing update from remote peer bf36e47a4a8b42eeb0928d0f58180602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:29.373446 22650 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:29.374393 22542 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [term 1 FOLLOWER]: Refusing update from remote peer bf36e47a4a8b42eeb0928d0f58180602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:29.375304 22650 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:29.377805 22685 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf36e47a4a8b42eeb0928d0f58180602. Latest consensus state: current_term: 1 leader_uuid: "bf36e47a4a8b42eeb0928d0f58180602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } } }
I20260926 20:20:29.377929 22685 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:29.379070 22672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf36e47a4a8b42eeb0928d0f58180602. Latest consensus state: current_term: 1 leader_uuid: "bf36e47a4a8b42eeb0928d0f58180602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } } }
I20260926 20:20:29.379215 22672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:29.388675 22685 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf36e47a4a8b42eeb0928d0f58180602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } } }
I20260926 20:20:29.388861 22685 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:29.389348 22702 mvcc.cc:205] Tried to move back new op lower bound from 7333699704304078848 to 7333699704242163712. Current Snapshot: MvccSnapshot[applied={T|T < 7333699704304078848}]
I20260926 20:20:29.391358 22650 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf36e47a4a8b42eeb0928d0f58180602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } } }
I20260926 20:20:29.391510 22650 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:29.394563 22684 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf36e47a4a8b42eeb0928d0f58180602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } } }
I20260926 20:20:29.394737 22684 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:29.394912 22701 mvcc.cc:205] Tried to move back new op lower bound from 7333699704304078848 to 7333699704242163712. Current Snapshot: MvccSnapshot[applied={T|T < 7333699704304078848}]
I20260926 20:20:29.395372 22672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf36e47a4a8b42eeb0928d0f58180602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36e47a4a8b42eeb0928d0f58180602" member_type: VOTER last_known_addr { host: "127.31.119.254" port: 37717 } } peers { permanent_uuid: "1fc5f08f36a24676a209c97ab19b3398" member_type: VOTER last_known_addr { host: "127.31.119.253" port: 40849 } } peers { permanent_uuid: "cf962ad56f654f52b9eca7149ef973d4" member_type: VOTER last_known_addr { host: "127.31.119.252" port: 36185 } } }
I20260926 20:20:29.395507 22672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:20:29.395643 22704 mvcc.cc:205] Tried to move back new op lower bound from 7333699704304078848 to 7333699704242163712. Current Snapshot: MvccSnapshot[applied={T|T < 7333699704304078848}]
I20260926 20:20:29.397814 22687 catalog_manager.cc:1347] Generated new cluster ID: 8aaca504b0404b56a97338c48feaad5f
I20260926 20:20:29.397879 22687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:29.430864 22687 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:29.432999 22687 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:29.454996 22687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf36e47a4a8b42eeb0928d0f58180602: Generated new TSK 0
I20260926 20:20:29.456112 22687 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:20:29.494169 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.490832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.31.119.253:40849)
0926 20:20:29.490963 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.490968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.492710 (+ 1742us) server_negotiation.cc:407] Connection header received
0926 20:20:29.493183 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.493187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.493242 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.493320 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.494016 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50410: BlockingRecv error: recv got EOF from 127.0.0.1:50410 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 20:20:29.495806 22497 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.491695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.31.119.254:37717)
0926 20:20:29.491806 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.491811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.494846 (+ 3035us) server_negotiation.cc:407] Connection header received
0926 20:20:29.495024 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.495029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.495084 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.495137 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.495690 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36408: BlockingRecv error: recv got EOF from 127.0.0.1:36408 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 20:20:29.497434 22648 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.492353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.31.119.252:36185)
0926 20:20:29.492472 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.492476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.496364 (+ 3888us) server_negotiation.cc:407] Connection header received
0926 20:20:29.496635 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.496638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.496695 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.496749 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.497268 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43248: BlockingRecv error: recv got EOF from 127.0.0.1:43248 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 20:20:29.501577 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.499875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.31.119.253:40849)
0926 20:20:29.499992 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.499997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.500522 (+ 525us) server_negotiation.cc:407] Connection header received
0926 20:20:29.500748 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.500751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.500834 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.500889 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.501465 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50422: BlockingRecv error: recv got EOF from 127.0.0.1:50422 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 20:20:29.507454 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.505781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.31.119.253:40849)
0926 20:20:29.505892 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.505896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.506440 (+ 544us) server_negotiation.cc:407] Connection header received
0926 20:20:29.506635 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.506639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.506695 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.506749 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.507337 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50438: BlockingRecv error: recv got EOF from 127.0.0.1:50438 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 20:20:29.513615 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.512063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.31.119.253:40849)
0926 20:20:29.512214 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.512218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.512559 (+ 341us) server_negotiation.cc:407] Connection header received
0926 20:20:29.512766 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.512770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.512825 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.512880 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.513473 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50440: BlockingRecv error: recv got EOF from 127.0.0.1:50440 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 20:20:29.521910 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.520025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.31.119.253:40849)
0926 20:20:29.520193 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.520197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.520809 (+ 612us) server_negotiation.cc:407] Connection header received
0926 20:20:29.521035 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.521038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.521097 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.521166 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.521793 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50444: BlockingRecv error: recv got EOF from 127.0.0.1:50444 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 20:20:29.534725 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.532644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.31.119.253:40849)
0926 20:20:29.532820 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.532825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.533625 (+ 800us) server_negotiation.cc:407] Connection header received
0926 20:20:29.533850 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.533853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.533911 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.533968 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.534601 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50454: BlockingRecv error: recv got EOF from 127.0.0.1:50454 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 20:20:29.555511 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.553490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.31.119.253:40849)
0926 20:20:29.553661 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.553665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.554402 (+ 737us) server_negotiation.cc:407] Connection header received
0926 20:20:29.554612 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.554616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.554675 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.554771 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.555379 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50458: BlockingRecv error: recv got EOF from 127.0.0.1:50458 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 20:20:29.591552 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.590190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.31.119.253:40849)
0926 20:20:29.590330 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.590335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.590612 (+ 277us) server_negotiation.cc:407] Connection header received
0926 20:20:29.590778 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.590781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.590840 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.590930 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.591420 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50462: BlockingRecv error: recv got EOF from 127.0.0.1:50462 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 20:20:29.659360 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.657990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.31.119.253:40849)
0926 20:20:29.658170 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.658175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.658362 (+ 187us) server_negotiation.cc:407] Connection header received
0926 20:20:29.658559 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.658563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.658617 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.658705 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.659217 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50470: BlockingRecv error: recv got EOF from 127.0.0.1:50470 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 20:20:29.790850 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:29.789721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.31.119.253:40849)
0926 20:20:29.789913 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:29.789916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:29.790054 (+ 138us) server_negotiation.cc:407] Connection header received
0926 20:20:29.790164 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:29.790167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:29.790218 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:29.790306 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:29.790748 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50484: BlockingRecv error: recv got EOF from 127.0.0.1:50484 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260926 20:20:30.050370 22572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:30.049175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.31.119.253:40849)
0926 20:20:30.049393 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:30.049400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:30.049433 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:20:30.049619 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:30.049622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:30.049678 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:30.049767 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:30.050237 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50500: BlockingRecv error: recv got EOF from 127.0.0.1:50500 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:20:30.369205 22683 catalog_manager.cc:1259] Loaded cluster ID: 8aaca504b0404b56a97338c48feaad5f
I20260926 20:20:30.369451 22683 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398: loading cluster ID for follower catalog manager: success
I20260926 20:20:30.371186 22683 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398: acquiring CA information for follower catalog manager: success
I20260926 20:20:30.371572 22698 catalog_manager.cc:1259] Loaded cluster ID: 8aaca504b0404b56a97338c48feaad5f
I20260926 20:20:30.371649 22698 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4: loading cluster ID for follower catalog manager: success
I20260926 20:20:30.372545 22683 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1fc5f08f36a24676a209c97ab19b3398: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:20:30.373387 22698 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4: acquiring CA information for follower catalog manager: success
I20260926 20:20:30.374538 22698 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf962ad56f654f52b9eca7149ef973d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:20:30.569650 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:30.564675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.31.119.253:40849)
0926 20:20:30.564979 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:30.564983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:30.565116 (+ 133us) server_negotiation.cc:407] Connection header received
0926 20:20:30.565322 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:30.565327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:30.565462 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:30.565567 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:30.565983 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:30.566641 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:30.567742 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:30.568164 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:30.568300 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:30.568670 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:30.569142 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:30.569322 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:30.569394 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 20:20:30.569456 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":124,"threads_started":1}
I20260926 20:20:30.577841 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:30.573650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.31.119.252:36185)
0926 20:20:30.573911 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:30.573916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:30.573931 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:20:30.574081 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:30.574085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:30.574222 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:30.574361 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:30.574600 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:30.575200 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:30.576129 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:30.576415 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:30.576533 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:30.576880 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:30.577413 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:30.577558 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:30.577596 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 20:20:30.577668 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
I20260926 20:20:30.584968 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:30.580727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.31.119.253:40849)
0926 20:20:30.580872 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:30.580876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:30.581365 (+ 489us) server_negotiation.cc:407] Connection header received
0926 20:20:30.581488 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:30.581491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:30.581561 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:30.581644 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:30.581888 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:30.582629 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:30.583793 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:30.584067 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:30.584177 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:30.584467 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:30.584572 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:30.584648 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:30.584754 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:20:30.584823 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:20:30.585217 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:30.581132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.31.119.252:36185)
0926 20:20:30.581227 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:30.581230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:30.581633 (+ 403us) server_negotiation.cc:407] Connection header received
0926 20:20:30.581721 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:30.581724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:30.581773 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:30.581887 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:30.582159 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:30.582862 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:30.583880 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:30.584214 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:30.584328 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:30.584582 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:30.584683 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:30.584764 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:30.585029 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 20:20:30.585081 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:20:30.586069 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:30.580716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.31.119.254:37717)
0926 20:20:30.580959 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:30.580964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:30.581332 (+ 368us) server_negotiation.cc:407] Connection header received
0926 20:20:30.581847 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:30.581854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:30.582021 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:30.582151 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:30.582505 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:30.583223 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:30.584359 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:30.584692 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:30.584778 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:30.585193 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:30.585616 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:30.585718 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:30.585853 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 20:20:30.585921 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20260926 20:20:30.589484 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 22425
I20260926 20:20:30.596298 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 22500
I20260926 20:20:30.602358 32223 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYRmynJ/build/debug/bin/kudu with pid 22574
2026-09-26T20:20:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3131 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6403 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (292623 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST